incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Donna Li" <donna...@utstar.com>
Subject gossiper problem
Date Fri, 15 Jul 2011 01:51:10 GMT
 

The following log is GC log of 10.63.61.71, 10.63.61.72-10.63.61.74 have no GC on date 2011-07-13.


10.63.61.71 and 10.63.61.72 are seed nodes.

My cassandra version is 0.6.3, and the configuration about gc on storage_conf.xml is 

<GCGraceSeconds>864000</GCGraceSeconds>

 

JVM configuration is as following:

JVM_OPTS=" \

        -ea \

        -Xms256M \

        -Xmx1G \

        -XX:+UseParNewGC \

        -XX:+UseConcMarkSweepGC \

        -XX:+CMSParallelRemarkEnabled \

        -XX:SurvivorRatio=8 \

        -XX:MaxTenuringThreshold=1 \

        -XX:+HeapDumpOnOutOfMemoryError \

        -Dcom.sun.management.jmxremote.port=8080 \

        -Dcom.sun.management.jmxremote.ssl=false \

        -Dcom.sun.management.jmxremote.authenticate=false"

 

Can I decrease the JVM_OPTS to –Xms128M –Xmx512M to avoid swap, the data saved in cassandra
is small, I do not need so much memory.

 

INFO [GC inspection] 2011-07-13 03:12:06,153 GCInspector.java (line 110) GC for ConcurrentMarkSweep:
1097 ms, 371528920 reclaimed leaving 17677528 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 03:12:07,351 GCInspector.java (line 110) GC for ParNew: 466
ms, 20619976 reclaimed leaving 157240232 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 03:25:54,378 GCInspector.java (line 110) GC for ParNew: 283
ms, 26850072 reclaimed leaving 154180424 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 06:29:58,092 GCInspector.java (line 110) GC for ParNew: 538
ms, 17358792 reclaimed leaving 222959936 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 09:48:33,379 GCInspector.java (line 110) GC for ParNew: 547
ms, 22483632 reclaimed leaving 291693248 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 10:44:57,435 GCInspector.java (line 110) GC for ParNew: 1621
ms, 9916072 reclaimed leaving 402812912 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 10:45:14,598 GCInspector.java (line 110) GC for ConcurrentMarkSweep:
12561 ms, 382596784 reclaimed leaving 17056608 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 10:50:24,947 GCInspector.java (line 110) GC for ParNew: 1186
ms, 23912976 reclaimed leaving 153561960 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 11:50:50,150 GCInspector.java (line 110) GC for ConcurrentMarkSweep:
253 ms, 403457896 reclaimed leaving 16975080 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 12:56:27,178 GCInspector.java (line 110) GC for ParNew: 359
ms, 21478776 reclaimed leaving 390794352 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 14:33:24,143 GCInspector.java (line 110) GC for ParNew: 658
ms, 26563640 reclaimed leaving 93002800 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 14:39:52,280 GCInspector.java (line 110) GC for ParNew: 449
ms, 26268648 reclaimed leaving 93554288 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 14:56:37,596 GCInspector.java (line 110) GC for ParNew: 1247
ms, 14271008 reclaimed leaving 239807832 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 15:14:47,254 GCInspector.java (line 110) GC for ParNew: 303
ms, 3355464 reclaimed leaving 408815760 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 15:14:47,255 GCInspector.java (line 110) GC for ConcurrentMarkSweep:
11719 ms, 387618656 reclaimed leaving 21197104 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 15:56:57,674 GCInspector.java (line 110) GC for ParNew: 4566
ms, 21030184 reclaimed leaving 166815416 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 16:03:24,392 GCInspector.java (line 110) GC for ParNew: 254
ms, 22468136 reclaimed leaving 305100176 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 16:03:37,105 GCInspector.java (line 110) GC for ParNew: 3953
ms, 18264560 reclaimed leaving 344948328 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 16:08:50,476 GCInspector.java (line 110) GC for ParNew: 2038
ms, 18900864 reclaimed leaving 416409032 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 16:09:29,023 GCInspector.java (line 110) GC for ParNew: 347
ms, 27133904 reclaimed leaving 52726592 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 16:15:21,151 GCInspector.java (line 110) GC for ParNew: 250
ms, 26106864 reclaimed leaving 53400456 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 17:02:33,948 GCInspector.java (line 110) GC for ParNew: 253
ms, 20448152 reclaimed leaving 226672208 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 17:38:34,250 GCInspector.java (line 110) GC for ParNew: 456
ms, 22461776 reclaimed leaving 363131080 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 18:02:24,432 GCInspector.java (line 110) GC for ParNew: 226
ms, 21273048 reclaimed leaving 156029352 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 18:08:25,259 GCInspector.java (line 110) GC for ParNew: 1077
ms, 8876672 reclaimed leaving 299731248 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 18:44:24,703 GCInspector.java (line 110) GC for ConcurrentMarkSweep:
449 ms, 380503216 reclaimed leaving 17038144 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:20:09,714 GCInspector.java (line 110) GC for ParNew: 353
ms, 26769688 reclaimed leaving 436912856 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:20:36,022 GCInspector.java (line 110) GC for ParNew: 7706
ms, 18728880 reclaimed leaving 478596160 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:21:41,233 GCInspector.java (line 110) GC for ParNew: 903
ms, 21769296 reclaimed leaving 516347064 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:21:41,246 GCInspector.java (line 110) GC for ConcurrentMarkSweep:
63440 ms, 495156848 reclaimed leaving 21190216 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:22:54,033 GCInspector.java (line 110) GC for ParNew: 232
ms, 26753464 reclaimed leaving 54874568 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:24:08,813 GCInspector.java (line 110) GC for ParNew: 1673
ms, 25808320 reclaimed leaving 123027728 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:24:16,498 GCInspector.java (line 110) GC for ParNew: 7515
ms, 11078160 reclaimed leaving 138793320 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:25:44,602 GCInspector.java (line 110) GC for ParNew: 222
ms, 26316512 reclaimed leaving 139425816 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:26:16,958 GCInspector.java (line 110) GC for ParNew: 265
ms, 27367784 reclaimed leaving 138927792 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:27:37,065 GCInspector.java (line 110) GC for ParNew: 264
ms, 26687704 reclaimed leaving 138982208 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:32:59,096 GCInspector.java (line 110) GC for ParNew: 1131
ms, 22081488 reclaimed leaving 244346808 used; max is 1187840000

 INFO [GC inspection] 2011-07-13 22:44:36,817 GCInspector.java (line 110) GC for ParNew: 380
ms, 22375728 reclaimed leaving 348001304 used; max is 1187840000

 INFO [GC inspection] 2011-07-14 00:26:27,381 GCInspector.java (line 110) GC for ParNew: 3116
ms, 9875432 reclaimed leaving 300588328 used; max is 1187840000

 INFO [GC inspection] 2011-07-14 00:44:30,918 GCInspector.java (line 110) GC for ParNew: 2434
ms, 12998784 reclaimed leaving 417469656 used; max is 1187840000

 INFO [GC inspection] 2011-07-14 01:26:35,149 GCInspector.java (line 110) GC for ConcurrentMarkSweep:
457 ms, 502581896 reclaimed leaving 17225448 used; max is 1187840000

 INFO [GC inspection] 2011-07-14 04:54:53,268 GCInspector.java (line 110) GC for ConcurrentMarkSweep:
278 ms, 407773688 reclaimed leaving 17250640 used; max is 1187840000

 INFO [GC inspection] 2011-07-14 07:08:25,240 GCInspector.java (line 110) GC for ParNew: 774
ms, 8645712 reclaimed leaving 368647304 used; max is 1187840000

 

Thanks

Donna li

________________________________

发件人: Vijay [mailto:vijay2win@gmail.com] 
发送时间: 2011年7月15日 9:27
收件人: user@cassandra.apache.org
主题: Re: gossiper problem

 

How about GC logs, what are your pause times? JVM settings might help....

If you are not sure how to enable GC logs check cassandra.yaml look for application pause
times. it is highly recommended not to swap -- include JNA jar.


Regards,
</VJ>




On Thu, Jul 14, 2011 at 1:42 AM, Donna Li <donna.li@utstar.com> wrote:

All:

I have four cassandra servers in cluster. I do not restart any one of the servers, why the
following print show the four servers restart many times? What is the possible reason? The
connection between the four server’s is good.

Swap may be used, because there are other applications run with cassandra server.

 

10.63.61.71 log

INFO [Timer-0] 2011-07-13 10:44:55,732 Gossiper.java (line 181) InetAddress /10.63.61.74 is
now dead.

 INFO [GMFD:1] 2011-07-13 10:44:57,748 Gossiper.java (line 579) InetAddress /10.63.61.74 is
now UP

 INFO [Timer-0] 2011-07-13 15:56:44,630 Gossiper.java (line 181) InetAddress /10.63.61.74
is now dead.

 INFO [GMFD:1] 2011-07-13 15:56:44,653 Gossiper.java (line 579) InetAddress /10.63.61.74 is
now UP

 INFO [Timer-0] 2011-07-13 16:03:24,391 Gossiper.java (line 181) InetAddress /10.63.61.72
is now dead.

 INFO [GMFD:1] 2011-07-13 16:03:24,405 Gossiper.java (line 579) InetAddress /10.63.61.72 is
now UP

 INFO [Timer-0] 2011-07-13 22:21:41,246 Gossiper.java (line 181) InetAddress /10.63.61.74
is now dead.

 INFO [Timer-0] 2011-07-13 22:22:45,602 Gossiper.java (line 181) InetAddress /10.63.61.73
is now dead.

 INFO [Timer-0] 2011-07-13 22:22:45,602 Gossiper.java (line 181) InetAddress /10.63.61.72
is now dead.

 INFO [GMFD:1] 2011-07-13 22:22:45,993 Gossiper.java (line 579) InetAddress /10.63.61.73 is
now UP

 INFO [GMFD:1] 2011-07-13 22:22:46,107 Gossiper.java (line 579) InetAddress /10.63.61.72 is
now UP

 INFO [GMFD:1] 2011-07-13 22:22:46,107 Gossiper.java (line 579) InetAddress /10.63.61.74 is
now UP

 INFO [Timer-0] 2011-07-13 22:24:08,812 Gossiper.java (line 181) InetAddress /10.63.61.74
is now dead.

 INFO [GMFD:1] 2011-07-13 22:24:08,920 Gossiper.java (line 579) InetAddress /10.63.61.74 is
now UP

 

10.63.61.72 log

INFO [Timer-0] 2011-07-13 02:06:03,941 Gossiper.java (line 181) InetAddress /10.63.61.71 is
now dead.

 INFO [GMFD:1] 2011-07-13 02:06:05,109 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 03:39:41,918 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 03:39:45,536 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 10:10:17,449 Gossiper.java (line 181) InetAddress /10.63.61.74
is now dead.

 INFO [Timer-0] 2011-07-13 10:10:17,471 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 10:10:18,451 Gossiper.java (line 579) InetAddress /10.63.61.74 is
now UP

 INFO [GMFD:1] 2011-07-13 10:10:18,451 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 10:44:36,140 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 10:44:57,417 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 10:45:10,141 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 10:45:14,478 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 15:14:44,044 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 15:14:47,610 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 15:56:36,857 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 15:56:44,417 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 16:02:37,260 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 16:02:52,651 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 16:03:05,289 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 16:03:11,260 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 16:08:47,666 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 16:08:48,668 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 17:38:32,569 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 17:38:34,572 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 22:20:45,706 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 22:22:46,143 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 22:23:32,875 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 22:24:08,948 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 22:32:37,421 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 22:32:38,036 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 

10.63.61.73 log

 INFO [Timer-0] 2011-07-13 03:39:42,066 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 03:39:45,536 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 10:42:37,274 Gossiper.java (line 181) InetAddress /10.63.61.74
is now dead.

 INFO [GMFD:1] 2011-07-13 10:42:37,275 Gossiper.java (line 579) InetAddress /10.63.61.74 is
now UP

 INFO [Timer-0] 2011-07-13 10:44:48,406 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 10:44:57,550 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 15:14:45,754 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 15:14:47,486 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 15:56:39,927 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 15:56:44,755 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 16:02:40,289 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 16:02:53,076 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 16:03:08,316 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 16:03:11,689 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 22:20:48,906 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 22:22:45,936 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 22:23:34,072 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 22:24:08,901 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 

10.63.61.74 log

 INFO [Timer-0] 2011-07-13 02:06:05,952 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 02:06:05,953 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 03:39:44,699 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 03:39:45,700 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 10:44:46,676 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 10:44:57,656 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 15:14:47,547 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 15:14:49,269 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 15:56:39,594 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 15:56:44,825 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 16:02:38,959 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 16:02:53,173 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 16:03:07,988 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 16:03:11,787 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 22:20:46,859 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 22:22:46,020 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 INFO [Timer-0] 2011-07-13 22:23:33,860 Gossiper.java (line 181) InetAddress /10.63.61.71
is now dead.

 INFO [GMFD:1] 2011-07-13 22:24:08,897 Gossiper.java (line 579) InetAddress /10.63.61.71 is
now UP

 

Best Regards

Donna li

 


Mime
View raw message