cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dimetrio <dimet...@flysoft.ru>
Subject GC eden filled instantly (any size). Dropping messages.
Date Mon, 27 Jan 2014 12:07:22 GMT
No one advice did't help to me for reduce GC load

I tried these:

MAX_HEAP_SIZE from default(8GB) to 16G with HEAP_NEWSIZE from 400M to 9600M
key cache on/off
compacting memory size and other limits
15 c3.4xlarge nodes (adding 5 nodes to 10 nodes cluster did't help): 
and many other

Reads ~5000 ops/s
Writes ~ 5000 ops/s
max batch is 50
heavy reads and heavy writes (and heavy deletes)
sometimes i have message: 
Read 1001 live and 2691 
Read 12 live and 2796


sudo jstat -gcutil -h15 `sudo cat /var/run/cassandra/cassandra.pid` 250ms 0
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
 18.93   0.00   4.52  75.36  59.77    225   30.119    18   28.361   58.480
  0.00  13.12   3.78  81.09  59.77    226   30.193    18   28.617   58.810
  0.00  13.12  39.50  81.09  59.78    226   30.193    18   28.617   58.810
  0.00  13.12  80.70  81.09  59.78    226   30.193    18   28.617   58.810
 17.21   9.13   0.66  87.38  59.78    228   30.235    18   28.617   58.852
  0.00  10.96  29.43  87.89  59.78    228   30.328    18   28.617   58.945
  0.00  10.96  62.67  87.89  59.78    228   30.328    18   28.617   58.945
  0.00  10.96  96.62  87.89  59.78    228   30.328    18   28.617   58.945
  0.00  10.69  10.29  94.56  59.78    230   30.462    18   28.617   59.078
  0.00  10.69  38.08  94.56  59.78    230   30.462    18   28.617   59.078
  0.00  10.69  71.70  94.56  59.78    230   30.462    18   28.617   59.078
 15.91   6.24   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123
 15.91   8.02   0.03  99.96  59.78    232   30.506    18   28.617   59.123


$ nodetool cfhistograms Social home_timeline
Social/home_timeline histograms
Offset      SSTables     Write Latency      Read Latency    Partition Size       
Cell Count
                              (micros)          (micros)           (bytes)
1              10458                 0                 0                 0            
26330
2              72428                 0                 0                 0               

0
3             339490                11                 0                 0            
42398
4             661819               156                 0                 0               

0
5              67186               893                 0                 0               

0
6              33284              3064                 0                 0            
15907
7              41287             10542                 0                 0               

0
8              49085             34689                 0                 0               

0
10             82941            244013                 0                 0            
19068
12             49438            523284                 0                 0            
17726
14             45109            724026                 0                 0               

0
17             69144           1181873                 0                 0            
18888
20             52563           1039934                 0                 0             
9859
24             63041           1007711                 0                 0            
16223
29             53042            702118                34                 0             
6289
35             31725            395098                78                 0            
10434
42             25571            212531                81                 0            
13735
50             11984            120692                69                 0             
8020
60              3985             71142                51                 0            
12874
72              4681             44988                52                 0            
10466
86                 0             31096                50             12336             
9162
103                0             25435                41                 0            
11421
124                0             20666                35                 0            
11204
149                0             17810                42              4799             
9953
179                0             18568                45             22462            
12347
215                0             20129                89              1492            
12628
258                0             26381               174              6605            
13687
310                0             34731               412             19060            
12978
372                0             58609               577              3848            
17080
446                0             59318              1355              9504            
13231
535                0             16462              2778             10685            
13791
642                0              9985              6555             13931            
12703
770                0              6403             14811             16859            
11261
924                0              4149             34059             12032            
12311
1109               0              3910             77621              8868            
11829
1331               0              3617            154224             12096            
12886
1597               0              3400            234929             10006            
11738
1916               0              3156            264185              9443            
11546
2299               0              2862            199240             10557            
12011
2759               0              2548            145502             10370            
12039
3311               0              1809            125259             10600            
14861
3973               0              1401             83855             10081            
13866
4768               0              1035             69407             10028            
13341
5722               0               795             55982             10594            
12707
6866               0               688             47374             10450            
11429
8239               0               338             37722             10608            
10324
9887               0               277             34395             12180             
9418
11864              0               255             31126             12896             
8542
14237              0               223             25607             12829             
7796
17084              0               178             20661             15812             
6803
20501              0               193             15916             15590             
6128
24601              0               198             12606             14920             
5290
29521              0               186             11137             14068             
4598
35425              0               195             11111             14377             
3891
42510              0               138              9597             14468             
3439
51012              0               123              7032             14298             
2701
61214              0               413              5288             15194             
2333
73457              0              1911              3991             13549             
1942
88148              0               940              3392             13178             
1595
105778             0               427              3045             12835             
1234
126934             0               431              2691             12676             
1080
152321             0               987              2504             15068              
787
182785             0               484              2337             14566              
600
219342             0               595              2052             13664              
516
263210             0               480              1736             13538              
377
315852             0               428              1428             12007              
316
379022             0               367              1093             10802              
240
454826             0               330               816              9819              
172
545791             0               285               555              8931              
155
654949             0               240               339              8153              
121
785939             0               168               201              7212               
99
943127             0               154               103              6477               
64
1131752            0               141                89              5693               
65
1358102            0               135                87              4885               
51
1629722            0                97                66              4210               
42
1955666            0                88                18              3689               
38
2346799            0                78                 7              3027               
18
2816159            0                73                21              2528               
23
3379391            0               120                54              2104               
11
4055269            0                70               107              1797               

7
4866323            0               118                37              1429               

3
5839588            0                56                73              1155               

3
7007506            0                31                52               895               

4
8409007            0                36                52               678               

3
10090808           0                29                 4               614               

4
12108970           0                34                17               402               

2
14530764           0                31                 7               369               

3
17436917           0                22                47               229               

2
20924300           0                20                 1               195               

0
25109160           0                34                36               154               

1


I'm at an impasse.



--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/GC-eden-filled-instantly-any-size-Dropping-messages-tp7592447.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Mime
View raw message