incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Narendra Sharma <narendra.sha...@gmail.com>
Subject Re: Pauses of GC
Date Thu, 17 Mar 2011 16:45:06 GMT
What heap size are you running with? and Which version of Cassandra?

Thanks,
Naren

On Thu, Mar 17, 2011 at 3:45 AM, ruslan usifov <ruslan.usifov@gmail.com>wrote:

> Hello
>
> Some times i have very long GC pauses:
>
>
> Total time for which application threads were stopped: 0.0303150 seconds
> 2011-03-17T13:19:56.476+0300: 33295.671: [GC 33295.671: [ParNew:
> 678855K->20708K(737280K), 0.0271230 secs] 1457643K->806795K(4112384K),
> 0.027305
> 0 secs] [Times: user=0.33 sys=0.00, real=0.03 secs]
> Total time for which application threads were stopped: 0.0291820 seconds
> 2011-03-17T13:20:32.962+0300: 33332.157: [GC 33332.157: [ParNew:
> 676068K->23527K(737280K), 0.0302180 secs] 1462155K->817599K(4112384K),
> 0.030402
> 0 secs] [Times: user=0.31 sys=0.00, real=0.03 secs]
> Total time for which application threads were stopped: 0.1270270 seconds
> 2011-03-17T13:21:11.908+0300: 33371.103: [GC 33371.103: [ParNew:
> 678887K->21564K(737280K), 0.0268160 secs] 1472959K->823191K(4112384K),
> 0.027011
> 0 secs] [Times: user=0.28 sys=0.00, real=0.03 secs]
> Total time for which application threads were stopped: 0.0293330 seconds
> 2011-03-17T13:21:50.482+0300: 33409.677: [GC 33409.677: [ParNew:
> 676924K->21115K(737280K), 0.0281720 secs] 1478551K->829900K(4112384K),
> 0.028363
> 0 secs] [Times: user=0.27 sys=0.00, real=0.03 secs]
> Total time for which application threads were stopped: 0.0339610 seconds
> 2011-03-17T13:22:32.849+0300: 33452.044: [GC 33452.044: [ParNew:
> 676475K->25948K(737280K), 0.0317600 secs] 1485260K->842061K(4112384K),
> 0.031952
> 0 secs] [Times: user=0.22 sys=0.00, real=0.03 secs]
> Total time for which application threads were stopped: 0.0344430 seconds
> 2011-03-17T13:23:14.924+0300: 33494.119: [GC 33494.119: [ParNew:
> 681308K->25087K(737280K), 0.0282600 secs] 1497421K->848300K(4112384K),
> 0.028436
> 0 secs] [Times: user=0.32 sys=0.00, real=0.03 secs]
> Total time for which application threads were stopped: 0.0309160 seconds
> 2011-03-17T13:23:57.192+0300: 33536.387: [GC 33536.387: [ParNew:
> 680447K->24805K(737280K), 0.0299910 secs] 1503660K->855829K(4112384K),
> 0.030167
> 0 secs] [Times: user=0.29 sys=0.01, real=0.03 secs]
> Total time for which application threads were stopped: 0.0324200 seconds
> 2011-03-17T13:24:01.553+0300: 33540.748: [GC 33540.749: [ParNew:
> 680165K->31886K(737280K), 0.0495620 secs] 1511189K->936503K(4112384K),
> 0.049742
> 0 secs] [Times: user=0.57 sys=0.00, real=0.05 secs]
> Total time for which application threads were stopped: 0.0507030 seconds
> 2011-03-17T13:37:56.009+0300: 34375.204: [GC 34375.204: [ParNew:
> 687246K->28727K(737280K), 0.0244720 secs] 1591863K->942459K(4112384K),
> 0.024690
> 0 secs] [Times: user=0.18 sys=0.00, real=0.02 secs]
> Total time for which application threads were stopped: 806.7442720 seconds
> Total time for which application threads were stopped: 0.0006590 seconds
> Total time for which application threads were stopped: 0.0004360 seconds
> Total time for which application threads were stopped: 0.0004630 seconds
> Total time for which application threads were stopped: 0.0008120 seconds
> 2011-03-17T13:37:59.018+0300: 34378.213: [GC 34378.213: [ParNew:
> 676678K->21640K(737280K), 0.0137740 secs] 1590410K->949991K(4112384K),
> 0.013961
> 0 secs] [Times: user=0.13 sys=0.02, real=0.01 secs]
> Total time for which application threads were stopped: 0.0145920 seconds
> Total time for which application threads were stopped: 0.1036080 seconds
> Total time for which application threads were stopped: 0.0585600 seconds
> Total time for which application threads were stopped: 0.0600550 seconds
> Total time for which application threads were stopped: 0.0008560 seconds
> Total time for which application threads were stopped: 0.0006770 seconds
> Total time for which application threads were stopped: 0.0005910 seconds
> Total time for which application threads were stopped: 0.0351330 seconds
> Total time for which application threads were stopped: 0.0329020 seconds
> Total time for which application threads were stopped: 0.0728490 seconds
> Total time for which application threads were stopped: 0.0480990 seconds
> Total time for which application threads were stopped: 0.0804250 seconds
> 2011-03-17T13:38:04.394+0300: 34383.589: [GC 34383.589: [ParNew:
> 677000K->8375K(737280K), 0.0218310 secs] 1605351K->944271K(4112384K),
> 0.0220300
>  secs]
>
>
>
>
> I have follow nodetoll cfstats on hung node:
>
> Keyspace: fishdom_tuenti
>     Read Count: 4970999
>     Read Latency: 1.0267005945887335 ms.
>     Write Count: 1441619
>     Write Latency: 0.013146585887117193 ms.
>     Pending Tasks: 0
>         Column Family: decor
>         SSTable count: 3
>         Space used (live): 1296203532
>         Space used (total): 1302520037
>         Memtable Columns Count: 1066
>         Memtable Data Size: 121742
>         Memtable Switch Count: 11
>         Read Count: 108125
>         Read Latency: 2.809 ms.
>         Write Count: 11261
>         Write Latency: 0.006 ms.
>         Pending Tasks: 0
>         Key cache capacity: 300000
>         Key cache size: 46470
>         Key cache hit rate: 0.40384615384615385
>         Row cache: disabled
>         Compacted row minimum size: 36
>         Compacted row maximum size: 73457
>         Compacted row mean size: 958
>
>         Column Family: adopt
>         SSTable count: 1
>         Space used (live): 3285283
>         Space used (total): 6585649
>         Memtable Columns Count: 14
>         Memtable Data Size: 322
>         Memtable Switch Count: 19
>         Read Count: 208
>         Read Latency: 0.009 ms.
>         Write Count: 248
>         Write Latency: 0.005 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 171
>         Key cache hit rate: NaN
>         Row cache: disabled
>         Compacted row minimum size: 87
>         Compacted row maximum size: 103
>         Compacted row mean size: 103
>
>         Column Family: user_store
>         SSTable count: 0
>         Space used (live): 0
>         Space used (total): 0
>         Memtable Columns Count: 0
>         Memtable Data Size: 0
>         Memtable Switch Count: 0
>         Read Count: 0
>         Read Latency: NaN ms.
>         Write Count: 0
>         Write Latency: NaN ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 0
>         Key cache hit rate: NaN
>         Row cache: disabled
>         Compacted row minimum size: 0
>         Compacted row maximum size: 0
>         Compacted row mean size: 0
>
>         Column Family: gifts
>         SSTable count: 1
>         Space used (live): 8767135
>         Space used (total): 17581781
>         Memtable Columns Count: 9
>         Memtable Data Size: 5993
>         Memtable Switch Count: 11
>         Read Count: 230
>         Read Latency: NaN ms.
>         Write Count: 306
>         Write Latency: 0.004 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 186
>         Key cache hit rate: NaN
>         Row cache: disabled
>         Compacted row minimum size: 125
>         Compacted row maximum size: 14237
>         Compacted row mean size: 498
>
>         Column Family: user_awards
>         SSTable count: 4
>         Space used (live): 285142885
>         Space used (total): 285142885
>         Memtable Columns Count: 399
>         Memtable Data Size: 21265
>         Memtable Switch Count: 11
>         Read Count: 602884
>         Read Latency: 0.075 ms.
>         Write Count: 3741
>         Write Latency: 0.003 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 9692
>         Key cache hit rate: 0.9929742388758782
>         Row cache: disabled
>         Compacted row minimum size: 104
>         Compacted row maximum size: 770
>         Compacted row mean size: 262
>
>         Column Family: aquarium
>         SSTable count: 4
>         Space used (live): 945412589
>         Space used (total): 970196634
>         Memtable Columns Count: 10973
>         Memtable Data Size: 1805059
>         Memtable Switch Count: 11
>         Read Count: 737198
>         Read Latency: 0.419 ms.
>         Write Count: 259785
>         Write Latency: 0.011 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 68960
>         Key cache hit rate: 0.9497041420118343
>         Row cache: disabled
>         Compacted row minimum size: 87
>         Compacted row maximum size: 1916
>         Compacted row mean size: 539
>
>         Column Family: user_messages
>         SSTable count: 1
>         Space used (live): 4648148
>         Space used (total): 9311688
>         Memtable Columns Count: 23
>         Memtable Data Size: 2525
>         Memtable Switch Count: 19
>         Read Count: 8782
>         Read Latency: 0.012 ms.
>         Write Count: 246
>         Write Latency: 0.012 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 2707
>         Key cache hit rate: NaN
>         Row cache: disabled
>         Compacted row minimum size: 180
>         Compacted row maximum size: 9887
>         Compacted row mean size: 380
>
>         Column Family: user_stat
>         SSTable count: 4
>         Space used (live): 872814042
>         Space used (total): 872814042
>         Memtable Columns Count: 36852
>         Memtable Data Size: 902102
>         Memtable Switch Count: 11
>         Read Count: 1663028
>         Read Latency: 0.446 ms.
>         Write Count: 406925
>         Write Latency: 0.010 ms.
>         Pending Tasks: 0
>         Key cache capacity: 400000
>         Key cache size: 78611
>         Key cache hit rate: 0.9583333333333334
>         Row cache: disabled
>         Compacted row minimum size: 51
>         Compacted row maximum size: 770
>         Compacted row mean size: 516
>
>         Column Family: user_stat_friend
>         SSTable count: 4
>         Space used (live): 614446488
>         Space used (total): 614446488
>         Memtable Columns Count: 9288
>         Memtable Data Size: 250169
>         Memtable Switch Count: 11
>         Read Count: 237287
>         Read Latency: 1.147 ms.
>         Write Count: 116691
>         Write Latency: 0.006 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 39410
>         Key cache hit rate: 0.8387096774193549
>         Row cache: disabled
>         Compacted row minimum size: 61
>         Compacted row maximum size: 258
>         Compacted row mean size: 199
>
>         Column Family: aquarium_friend
>         SSTable count: 4
>         Space used (live): 7135867112
>         Space used (total): 7145863363
>         Memtable Columns Count: 6061
>         Memtable Data Size: 618699
>         Memtable Switch Count: 11
>         Read Count: 333994
>         Read Latency: 1.358 ms.
>         Write Count: 168818
>         Write Latency: 0.010 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 47973
>         Key cache hit rate: 0.8808664259927798
>         Row cache: disabled
>         Compacted row minimum size: 104
>         Compacted row maximum size: 924
>         Compacted row mean size: 304
>
>         Column Family: aquarium_messages
>         SSTable count: 1
>         Space used (live): 5949016
>         Space used (total): 11912616
>         Memtable Columns Count: 11
>         Memtable Data Size: 1355
>         Memtable Switch Count: 19
>         Read Count: 26441
>         Read Latency: 0.010 ms.
>         Write Count: 279
>         Write Latency: 0.003 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 4095
>         Key cache hit rate: NaN
>         Row cache: disabled
>         Compacted row minimum size: 180
>         Compacted row maximum size: 14237
>         Compacted row mean size: 382
>
>         Column Family: user_params
>         SSTable count: 0
>         Space used (live): 0
>         Space used (total): 0
>         Memtable Columns Count: 0
>         Memtable Data Size: 0
>         Memtable Switch Count: 0
>         Read Count: 0
>         Read Latency: NaN ms.
>         Write Count: 0
>         Write Latency: NaN ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 0
>         Key cache hit rate: NaN
>         Row cache: disabled
>         Compacted row minimum size: 0
>         Compacted row maximum size: 0
>         Compacted row mean size: 0
>
>         Column Family: gems
>         SSTable count: 4
>         Space used (live): 582170315
>         Space used (total): 582170315
>         Memtable Columns Count: 8713
>         Memtable Data Size: 285047
>         Memtable Switch Count: 11
>         Read Count: 97307
>         Read Latency: 0.733 ms.
>         Write Count: 35400
>         Write Latency: 0.009 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 21120
>         Key cache hit rate: 0.8691588785046729
>         Row cache: disabled
>         Compacted row minimum size: 36
>         Compacted row maximum size: 2759
>         Compacted row mean size: 691
>
>         Column Family: adopt_accept
>         SSTable count: 1
>         Space used (live): 3531249
>         Space used (total): 7077144
>         Memtable Columns Count: 15
>         Memtable Data Size: 433
>         Memtable Switch Count: 19
>         Read Count: 271
>         Read Latency: 0.011 ms.
>         Write Count: 249
>         Write Latency: 0.004 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 325
>         Key cache hit rate: NaN
>         Row cache: disabled
>         Compacted row minimum size: 87
>         Compacted row maximum size: 179
>         Compacted row mean size: 109
>
>         Column Family: gems_friend
>         SSTable count: 5
>         Space used (live): 1620588533
>         Space used (total): 1620588533
>         Memtable Columns Count: 21441
>         Memtable Data Size: 577929
>         Memtable Switch Count: 11
>         Read Count: 265787
>         Read Latency: 0.974 ms.
>         Write Count: 135310
>         Write Latency: 0.011 ms.
>         Pending Tasks: 0
>         Key cache capacity: 200000
>         Key cache size: 49702
>         Key cache hit rate: 0.8641304347826086
>         Row cache: disabled
>         Compacted row minimum size: 36
>         Compacted row maximum size: 1916
>         Compacted row mean size: 536
>
>         Column Family: fish
>         SSTable count: 5
>         Space used (live): 3424896182
>         Space used (total): 3543066763
>         Memtable Columns Count: 88706
>         Memtable Data Size: 10660109
>         Memtable Switch Count: 11
>         Read Count: 889507
>         Read Latency: 0.718 ms.
>         Write Count: 302385
>         Write Latency: 0.024 ms.
>         Pending Tasks: 0
>         Key cache capacity: 400000
>         Key cache size: 90802
>         Key cache hit rate: 0.9509594882729211
>         Row cache: disabled
>         Compacted row minimum size: 51
>         Compacted row maximum size: 51012
>         Compacted row mean size: 1890
>
> ----------------
>

Mime
View raw message