incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ruslan usifov <ruslan.usi...@gmail.com>
Subject Pauses of GC
Date Thu, 17 Mar 2011 10:45:10 GMT
Hello

Some times i have very long GC pauses:


Total time for which application threads were stopped: 0.0303150 seconds
2011-03-17T13:19:56.476+0300: 33295.671: [GC 33295.671: [ParNew:
678855K->20708K(737280K), 0.0271230 secs] 1457643K->806795K(4112384K),
0.027305
0 secs] [Times: user=0.33 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0291820 seconds
2011-03-17T13:20:32.962+0300: 33332.157: [GC 33332.157: [ParNew:
676068K->23527K(737280K), 0.0302180 secs] 1462155K->817599K(4112384K),
0.030402
0 secs] [Times: user=0.31 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.1270270 seconds
2011-03-17T13:21:11.908+0300: 33371.103: [GC 33371.103: [ParNew:
678887K->21564K(737280K), 0.0268160 secs] 1472959K->823191K(4112384K),
0.027011
0 secs] [Times: user=0.28 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0293330 seconds
2011-03-17T13:21:50.482+0300: 33409.677: [GC 33409.677: [ParNew:
676924K->21115K(737280K), 0.0281720 secs] 1478551K->829900K(4112384K),
0.028363
0 secs] [Times: user=0.27 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0339610 seconds
2011-03-17T13:22:32.849+0300: 33452.044: [GC 33452.044: [ParNew:
676475K->25948K(737280K), 0.0317600 secs] 1485260K->842061K(4112384K),
0.031952
0 secs] [Times: user=0.22 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0344430 seconds
2011-03-17T13:23:14.924+0300: 33494.119: [GC 33494.119: [ParNew:
681308K->25087K(737280K), 0.0282600 secs] 1497421K->848300K(4112384K),
0.028436
0 secs] [Times: user=0.32 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0309160 seconds
2011-03-17T13:23:57.192+0300: 33536.387: [GC 33536.387: [ParNew:
680447K->24805K(737280K), 0.0299910 secs] 1503660K->855829K(4112384K),
0.030167
0 secs] [Times: user=0.29 sys=0.01, real=0.03 secs]
Total time for which application threads were stopped: 0.0324200 seconds
2011-03-17T13:24:01.553+0300: 33540.748: [GC 33540.749: [ParNew:
680165K->31886K(737280K), 0.0495620 secs] 1511189K->936503K(4112384K),
0.049742
0 secs] [Times: user=0.57 sys=0.00, real=0.05 secs]
Total time for which application threads were stopped: 0.0507030 seconds
2011-03-17T13:37:56.009+0300: 34375.204: [GC 34375.204: [ParNew:
687246K->28727K(737280K), 0.0244720 secs] 1591863K->942459K(4112384K),
0.024690
0 secs] [Times: user=0.18 sys=0.00, real=0.02 secs]
Total time for which application threads were stopped: 806.7442720 seconds
Total time for which application threads were stopped: 0.0006590 seconds
Total time for which application threads were stopped: 0.0004360 seconds
Total time for which application threads were stopped: 0.0004630 seconds
Total time for which application threads were stopped: 0.0008120 seconds
2011-03-17T13:37:59.018+0300: 34378.213: [GC 34378.213: [ParNew:
676678K->21640K(737280K), 0.0137740 secs] 1590410K->949991K(4112384K),
0.013961
0 secs] [Times: user=0.13 sys=0.02, real=0.01 secs]
Total time for which application threads were stopped: 0.0145920 seconds
Total time for which application threads were stopped: 0.1036080 seconds
Total time for which application threads were stopped: 0.0585600 seconds
Total time for which application threads were stopped: 0.0600550 seconds
Total time for which application threads were stopped: 0.0008560 seconds
Total time for which application threads were stopped: 0.0006770 seconds
Total time for which application threads were stopped: 0.0005910 seconds
Total time for which application threads were stopped: 0.0351330 seconds
Total time for which application threads were stopped: 0.0329020 seconds
Total time for which application threads were stopped: 0.0728490 seconds
Total time for which application threads were stopped: 0.0480990 seconds
Total time for which application threads were stopped: 0.0804250 seconds
2011-03-17T13:38:04.394+0300: 34383.589: [GC 34383.589: [ParNew:
677000K->8375K(737280K), 0.0218310 secs] 1605351K->944271K(4112384K),
0.0220300
 secs]




I have follow nodetoll cfstats on hung node:

Keyspace: fishdom_tuenti
    Read Count: 4970999
    Read Latency: 1.0267005945887335 ms.
    Write Count: 1441619
    Write Latency: 0.013146585887117193 ms.
    Pending Tasks: 0
        Column Family: decor
        SSTable count: 3
        Space used (live): 1296203532
        Space used (total): 1302520037
        Memtable Columns Count: 1066
        Memtable Data Size: 121742
        Memtable Switch Count: 11
        Read Count: 108125
        Read Latency: 2.809 ms.
        Write Count: 11261
        Write Latency: 0.006 ms.
        Pending Tasks: 0
        Key cache capacity: 300000
        Key cache size: 46470
        Key cache hit rate: 0.40384615384615385
        Row cache: disabled
        Compacted row minimum size: 36
        Compacted row maximum size: 73457
        Compacted row mean size: 958

        Column Family: adopt
        SSTable count: 1
        Space used (live): 3285283
        Space used (total): 6585649
        Memtable Columns Count: 14
        Memtable Data Size: 322
        Memtable Switch Count: 19
        Read Count: 208
        Read Latency: 0.009 ms.
        Write Count: 248
        Write Latency: 0.005 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 171
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 87
        Compacted row maximum size: 103
        Compacted row mean size: 103

        Column Family: user_store
        SSTable count: 0
        Space used (live): 0
        Space used (total): 0
        Memtable Columns Count: 0
        Memtable Data Size: 0
        Memtable Switch Count: 0
        Read Count: 0
        Read Latency: NaN ms.
        Write Count: 0
        Write Latency: NaN ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 0
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 0
        Compacted row maximum size: 0
        Compacted row mean size: 0

        Column Family: gifts
        SSTable count: 1
        Space used (live): 8767135
        Space used (total): 17581781
        Memtable Columns Count: 9
        Memtable Data Size: 5993
        Memtable Switch Count: 11
        Read Count: 230
        Read Latency: NaN ms.
        Write Count: 306
        Write Latency: 0.004 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 186
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 125
        Compacted row maximum size: 14237
        Compacted row mean size: 498

        Column Family: user_awards
        SSTable count: 4
        Space used (live): 285142885
        Space used (total): 285142885
        Memtable Columns Count: 399
        Memtable Data Size: 21265
        Memtable Switch Count: 11
        Read Count: 602884
        Read Latency: 0.075 ms.
        Write Count: 3741
        Write Latency: 0.003 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 9692
        Key cache hit rate: 0.9929742388758782
        Row cache: disabled
        Compacted row minimum size: 104
        Compacted row maximum size: 770
        Compacted row mean size: 262

        Column Family: aquarium
        SSTable count: 4
        Space used (live): 945412589
        Space used (total): 970196634
        Memtable Columns Count: 10973
        Memtable Data Size: 1805059
        Memtable Switch Count: 11
        Read Count: 737198
        Read Latency: 0.419 ms.
        Write Count: 259785
        Write Latency: 0.011 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 68960
        Key cache hit rate: 0.9497041420118343
        Row cache: disabled
        Compacted row minimum size: 87
        Compacted row maximum size: 1916
        Compacted row mean size: 539

        Column Family: user_messages
        SSTable count: 1
        Space used (live): 4648148
        Space used (total): 9311688
        Memtable Columns Count: 23
        Memtable Data Size: 2525
        Memtable Switch Count: 19
        Read Count: 8782
        Read Latency: 0.012 ms.
        Write Count: 246
        Write Latency: 0.012 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 2707
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 180
        Compacted row maximum size: 9887
        Compacted row mean size: 380

        Column Family: user_stat
        SSTable count: 4
        Space used (live): 872814042
        Space used (total): 872814042
        Memtable Columns Count: 36852
        Memtable Data Size: 902102
        Memtable Switch Count: 11
        Read Count: 1663028
        Read Latency: 0.446 ms.
        Write Count: 406925
        Write Latency: 0.010 ms.
        Pending Tasks: 0
        Key cache capacity: 400000
        Key cache size: 78611
        Key cache hit rate: 0.9583333333333334
        Row cache: disabled
        Compacted row minimum size: 51
        Compacted row maximum size: 770
        Compacted row mean size: 516

        Column Family: user_stat_friend
        SSTable count: 4
        Space used (live): 614446488
        Space used (total): 614446488
        Memtable Columns Count: 9288
        Memtable Data Size: 250169
        Memtable Switch Count: 11
        Read Count: 237287
        Read Latency: 1.147 ms.
        Write Count: 116691
        Write Latency: 0.006 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 39410
        Key cache hit rate: 0.8387096774193549
        Row cache: disabled
        Compacted row minimum size: 61
        Compacted row maximum size: 258
        Compacted row mean size: 199

        Column Family: aquarium_friend
        SSTable count: 4
        Space used (live): 7135867112
        Space used (total): 7145863363
        Memtable Columns Count: 6061
        Memtable Data Size: 618699
        Memtable Switch Count: 11
        Read Count: 333994
        Read Latency: 1.358 ms.
        Write Count: 168818
        Write Latency: 0.010 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 47973
        Key cache hit rate: 0.8808664259927798
        Row cache: disabled
        Compacted row minimum size: 104
        Compacted row maximum size: 924
        Compacted row mean size: 304

        Column Family: aquarium_messages
        SSTable count: 1
        Space used (live): 5949016
        Space used (total): 11912616
        Memtable Columns Count: 11
        Memtable Data Size: 1355
        Memtable Switch Count: 19
        Read Count: 26441
        Read Latency: 0.010 ms.
        Write Count: 279
        Write Latency: 0.003 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 4095
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 180
        Compacted row maximum size: 14237
        Compacted row mean size: 382

        Column Family: user_params
        SSTable count: 0
        Space used (live): 0
        Space used (total): 0
        Memtable Columns Count: 0
        Memtable Data Size: 0
        Memtable Switch Count: 0
        Read Count: 0
        Read Latency: NaN ms.
        Write Count: 0
        Write Latency: NaN ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 0
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 0
        Compacted row maximum size: 0
        Compacted row mean size: 0

        Column Family: gems
        SSTable count: 4
        Space used (live): 582170315
        Space used (total): 582170315
        Memtable Columns Count: 8713
        Memtable Data Size: 285047
        Memtable Switch Count: 11
        Read Count: 97307
        Read Latency: 0.733 ms.
        Write Count: 35400
        Write Latency: 0.009 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 21120
        Key cache hit rate: 0.8691588785046729
        Row cache: disabled
        Compacted row minimum size: 36
        Compacted row maximum size: 2759
        Compacted row mean size: 691

        Column Family: adopt_accept
        SSTable count: 1
        Space used (live): 3531249
        Space used (total): 7077144
        Memtable Columns Count: 15
        Memtable Data Size: 433
        Memtable Switch Count: 19
        Read Count: 271
        Read Latency: 0.011 ms.
        Write Count: 249
        Write Latency: 0.004 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 325
        Key cache hit rate: NaN
        Row cache: disabled
        Compacted row minimum size: 87
        Compacted row maximum size: 179
        Compacted row mean size: 109

        Column Family: gems_friend
        SSTable count: 5
        Space used (live): 1620588533
        Space used (total): 1620588533
        Memtable Columns Count: 21441
        Memtable Data Size: 577929
        Memtable Switch Count: 11
        Read Count: 265787
        Read Latency: 0.974 ms.
        Write Count: 135310
        Write Latency: 0.011 ms.
        Pending Tasks: 0
        Key cache capacity: 200000
        Key cache size: 49702
        Key cache hit rate: 0.8641304347826086
        Row cache: disabled
        Compacted row minimum size: 36
        Compacted row maximum size: 1916
        Compacted row mean size: 536

        Column Family: fish
        SSTable count: 5
        Space used (live): 3424896182
        Space used (total): 3543066763
        Memtable Columns Count: 88706
        Memtable Data Size: 10660109
        Memtable Switch Count: 11
        Read Count: 889507
        Read Latency: 0.718 ms.
        Write Count: 302385
        Write Latency: 0.024 ms.
        Pending Tasks: 0
        Key cache capacity: 400000
        Key cache size: 90802
        Key cache hit rate: 0.9509594882729211
        Row cache: disabled
        Compacted row minimum size: 51
        Compacted row maximum size: 51012
        Compacted row mean size: 1890

----------------

Mime
View raw message