cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xutom <xutom2...@126.com>
Subject cassandra full gc too long
Date Tue, 29 Dec 2015 01:57:27 GMT
Hi all,
    I have 5 nodes in my C* cluster, and each node has the same configuration file(Cassandra-env.sh:
MAX_HEAP_SIZE="32G" and HEAP_NEWSIZE="8G"), and My Cassandra version is 2.1.1. Now I want
to export all data of one table, i am using  select * from tablename, and set the bStatement.setFetchSize(10000);
When exporting the data, I check the status of the C* cluster, I found the following:
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns    Host ID                               Rack
UN  134.13.13.36  15.82 GB   256     ?       67aecea2-00f5-4001-ba4c-dc6138edaaa0  rack1
UN  134.13.13.37  20.25 GB   256     ?       a8c0ec24-6e48-4082-be53-32a971a4aaaa  rack1
DN  134.13.13.33  18.36 GB   256     ?       d93f8201-a3a7-404a-8dd6-164ddbfeaaa7  rack1
UN  134.13.13.34  16.9 GB    256     ?       6880cf40-0b73-4bbc-a0a5-7799a983aaab  rack1
UN  134.13.13.35  19.78 GB   256     ?       5658ae76-40a0-441f-908e-9d11ab2baaad  rack1
The Cassandra jstat information on node 134.13.13.33 is following. I can see that Cassandra
does not use S1, and it does not do the young GC, instead, it just do Full GC, and it do so
many Full GC and so long, so C* cluster think it is Down. The node 134.13.13.33 can only work
normally after startup for a while, after many times to do Full GC, it is Down and cannot
finish the Full GC progress and crash at the end.
[cache@cache02 caslog]$ jstat -gcutil 62335 4000 2000
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
 86.49   0.00 100.00  98.02  72.54     19   38.236     3    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.54     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.52     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.52     19   38.236     4    0.863   39.099
 86.49   0.00 100.00  98.02  72.52     19   38.236     4    0.863   39.099
  0.00   0.00  41.13 100.00  72.52     19   38.236     4   71.671  109.906
  0.00   0.00  41.13 100.00  72.52     19   38.236     4   71.671  109.906
  0.00   0.00  41.13 100.00  72.52     19   38.236     4   71.671  109.906
  0.00   0.00  41.13 100.00  72.52     19   38.236     4   71.671  109.906
  0.00   0.00  77.65 100.00  60.03     19   38.236     5   71.671  109.906
 20.27   0.00 100.00 100.00  60.27     19   38.236     5   73.905  112.141
 58.59   0.00 100.00 100.00  60.30     19   38.236     5   73.905  112.141
 98.19   0.00 100.00 100.00  60.30     19   38.236     5   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     5   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     5   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     5   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     5   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
100.00   0.00 100.00 100.00  60.30     19   38.236     6   73.905  112.141
  0.00   0.00  78.90 100.00  60.30     19   38.236     6  141.773  180.008
  0.00   0.00  78.90 100.00  60.30     19   38.236     6  141.773  180.008
  0.00   0.00  78.90 100.00  60.30     19   38.236     6  141.773  180.008


Other Cassandra jstat information of other node of this C* cluster is following, I think it
is normal because it can do young GC so many times then do Full GC:
[cache@cache06 ~]$ jstat -gcutil 32060 4000 2000     
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
  0.00 100.00  26.53  59.08  60.02    609  140.372    12   89.422  229.794
  0.00 100.00  33.48  61.20  60.02    611  140.834    12   89.422  230.257
  0.00 100.00  50.18  63.71  60.02    613  141.384    12   89.422  230.807
 11.65   0.00  83.51  66.31  60.02    616  141.723    12   89.422  231.146
100.00 100.00 100.00  68.40  60.02    619  142.167    12   89.422  231.590
  0.00 100.00  13.15  71.60  60.02    621  143.195    12   89.422  232.618
  0.00 100.00  27.69  73.89  60.02    623  143.679    12   89.422  233.102
100.00   0.00  71.57  74.60  60.02    624  143.772    12   89.422  233.194
  0.00  94.92  88.87  74.97  60.02    625  143.838    12   89.422  233.260
100.00 100.00 100.00  76.37  60.02    627  143.942    13   89.577  233.519
  0.00 100.00  78.16  76.32  60.02    627  144.027    14   90.273  234.300
  0.00 100.00  31.22  58.51  60.02    629  144.425    14   90.273  234.698
100.00   0.00  27.90  44.76  60.02    632  145.104    14   90.273  235.377
100.00   0.00  32.10  29.26  60.02    634  145.254    14   90.273  235.527
 83.50   0.00  65.04  27.60  59.78    636  145.395    14   90.273  235.669
100.00   0.00  72.60  28.20  59.78    638  145.534    14   90.273  235.807
  0.00 100.00   5.97  28.73  59.78    641  145.703    14   90.273  235.976
  0.00  76.34  34.28  29.32  59.78    643  145.877    14   90.273  236.150
  0.00  97.42  91.21  29.50  59.79    645  145.971    14   90.273  236.244
  0.00 100.00  94.70  29.72  59.79    647  146.070    14   90.273  236.343
 78.65   0.00  46.82  30.19  59.79    650  146.201    14   90.273  236.475
 59.25   0.00  49.98  30.19  59.79    652  146.267    14   90.273  236.540
Mime
View raw message