cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuo Chen <chenatu2...@gmail.com>
Subject Re: cassandra full gc too often
Date Mon, 04 Jan 2016 02:34:42 GMT
The following is the snippets of nodetool compactions history. It shows
many records relates to keyspace system.

Compaction History:
id                                       keyspace_name
 columnfamily_name            compacted_at              bytes_in
bytes_out      rows_merged
8e4f8830-b04f-11e5-a211-45b7aa88107c     system
sstable_activity             1451629144115             3342           915
         {4:23}
96a6fcb0-b04b-11e5-a211-45b7aa88107c     system             hints
             1451627440123             18970740       18970740       {1:1}
7c42c940-adac-11e5-8bd4-45b7aa88107c     system             hints
             1451339203540             56969835       56782732       {2:3}
585b97a0-ad98-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451330553370             3700           956
         {4:24}
aefc3f10-b1b2-11e5-a211-45b7aa88107c     system
sstable_activity             1451781670273             3201           906
         {4:23}
1e76f1b0-b180-11e5-a211-45b7aa88107c     system
sstable_activity             1451759952971             3303           700
         {4:23}
e7b75b70-aec2-11e5-8bd4-45b7aa88107c     system             hints
             1451458783911             57690316       57497847       {2:3}
ad102280-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451532129448
    45671877       41137664       {1:11, 3:1, 4:8}
60906970-aec7-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451460704647             3751           974
         {4:25}
88aed310-ad91-11e5-8bd4-45b7aa88107c     system             hints
             1451327627969             56984347       56765328       {2:3}
3ad14f00-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451531937776
    46696097       38827028       {1:8, 3:2, 4:9}
84df8fb0-b00f-11e5-a211-45b7aa88107c     system             hints
             1451601640491             18970740       18970740       {1:1}
657482e0-ad33-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451287196174             3701           931
         {4:24}
9cc8af70-b24a-11e5-a211-45b7aa88107c     system
sstable_activity             1451846923239             3134           773
         {4:23}
dcbe5e30-afd0-11e5-a211-45b7aa88107c     system
sstable_activity             1451574729619             3357           790
         {4:23}
b285ced0-afa0-11e5-84e3-45b7aa88107c     system             hints
             1451554042941             43310718       42137761       {1:1,
2:2}
119770e0-ad4e-11e5-8bd4-45b7aa88107c     system             hints
             1451298651886             57397441       57190519       {2:3}
f1bb37a0-b204-11e5-a211-45b7aa88107c     system             hints
             1451817000986             17713746       17566349       {1:1}
dcd0c720-b0b4-11e5-a211-45b7aa88107c     system
sstable_activity             1451672654993             3189           797
         {4:23}
cadb2320-ae94-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451438978642             3642           959
         {4:25}
321a84d0-af63-11e5-8bd4-45b7aa88107c     system             hints
             1451527628189             57407115       57179555       {1:3}
5d760ca0-af20-11e5-8bd4-45b7aa88107c     system             hints
             1451498924650             57405307       57234936       {1:3}
2e59a0b0-af6d-11e5-b1dc-45b7aa88107c     system
compactions_in_progress      1451531916859             464            190
         {1:1, 2:1}
8723a290-adba-11e5-8bd4-45b7aa88107c     system             hints
             1451345234745             57065795       56814910       {2:3}
add0a8d0-af6c-11e5-b1dc-45b7aa88107c     system
compactions_in_progress      1451531701213             464            42
          {2:2}
04c0d2f0-af6d-11e5-b1dc-45b7aa88107c     system
compactions_in_progress      1451531847071             446            0
         {2:2}
315dd170-af98-11e5-b1dc-45b7aa88107c     system             schema_columns
              1451550390279             129142         45841          {2:1,
3:1, 4:9}
a17d8f80-b24b-11e5-a211-45b7aa88107c     system             hints
             1451847360632             16746317       16600978       {1:1}
e22e3b90-ad39-11e5-8bd4-45b7aa88107c     system             hints
             1451289982409             57240038       56984895       {2:3}
38591c10-b22b-11e5-a211-45b7aa88107c     system             hints
             1451833440337             17351684       17002773       {1:1}
fc274930-af6c-11e5-b1dc-45b7aa88107c     system             hints
             1451531832643             63540861       59716559       {2:3}
aba057f0-b1b4-11e5-a211-45b7aa88107c     system             hints
             1451782523631             18743329       18577220       {1:1}
81d5e980-af5c-11e5-8bd4-45b7aa88107c     system             hints
             1451524755480             57603966       57407115       {2:3}
a1801510-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451532110049
    46735894       40839394       {1:11, 4:9}
f7e02540-b1bf-11e5-a211-45b7aa88107c     system             hints
             1451787376020             18577220       18403648       {1:1}
3e9a72b0-af72-11e5-b1dc-45b7aa88107c     system             hints
             1451534091611             60455761       43314155       {2:1,
3:2}
30c103c0-af6d-11e5-b1dc-45b7aa88107c     system             peers
             1451531920892             1396           567            {2:3,
3:1}
65f47770-aed7-11e5-8bd4-45b7aa88107c     system             hints
             1451467585639             57780382       57445632       {2:3}
ecabc460-afa6-11e5-a211-45b7aa88107c     system             hints
             1451556717478             29776619       29776619       {1:2}
5da4df30-afac-11e5-a211-45b7aa88107c     system             hints
             1451559054499             20309429       18970740       {1:1,
2:1}
f5ce2d30-afa6-11e5-a211-45b7aa88107c     system             hints
             1451556732803             29776619       29776619       {1:2}
4ada6610-aeec-11e5-8bd4-45b7aa88107c     system             hints
             1451476559601             57459788       57221330       {1:3}
74ffd7f0-ae32-11e5-8bd4-45b7aa88107c     system             hints
             1451396743919             57210071       57053060       {2:3}
621f05e0-b0ce-11e5-a211-45b7aa88107c     system
sstable_activity             1451683616062             3060           719
         {4:23}
5ec0e5b0-af2c-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451504080779             3756           971
         {4:25}
92707090-b09b-11e5-a211-45b7aa88107c     system
sstable_activity             1451661792793             3149           689
         {4:23}
c35b51f0-ae40-11e5-8bd4-45b7aa88107c     system             hints
             1451402888335             57190905       57037697       {2:3}
2a813e10-ade4-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451363118193             3772           960
         {4:24}
c99593a0-af6c-11e5-b1dc-45b7aa88107c     system             hints
             1451531747802             57645358       57613272       {3:3}
50002a00-aee5-11e5-8bd4-45b7aa88107c     system             hints
             1451473561760             57670199       57459788       {2:3}
c2cf2570-ae6a-11e5-8bd4-45b7aa88107c     system             hints
             1451420926279             57375776       57227330       {2:3}
1b5e7bc0-ad1a-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451276334460             3606           861
         {4:24}
bf2f8d90-adc7-11e5-8bd4-45b7aa88107c     system             hints
             1451350912233             56901257       56731829       {2:3}
55e055e0-b263-11e5-a211-45b7aa88107c     system             hints
             1451857541694             16600978       16364535       {1:1}
1a60f830-afa7-11e5-a211-45b7aa88107c     system             hints
             1451556794163             29776619       29776619       {1:2}
7b193990-adfd-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451373990825             3679           939
         {4:24}
6bf27720-aec9-11e5-8bd4-45b7aa88107c     system             hints
             1451461582737             57571883       57372615       {2:3}
d68f67b0-af13-11e5-8bd4-45b7aa88107c     system             hints
             1451493544363             57518024       57335312       {2:3}
7946c2a0-b162-11e5-a211-45b7aa88107c     system             hints
             1451747220426             18970740       18970740       {1:1}
1b686990-ad98-11e5-8bd4-45b7aa88107c     system             hints
             1451330451113             56988235       56854912       {2:3}
30883420-afa7-11e5-a211-45b7aa88107c     system             hints
             1451556831330             29776619       29776619       {1:2}
e5651290-b129-11e5-a211-45b7aa88107c     system             hints
             1451722920505             18970740       18970740       {1:1}
feedabd0-ad61-11e5-8bd4-45b7aa88107c     system             hints
             1451307210509             57251764       57133776       {2:3}
d75a36b0-afa0-11e5-84e3-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451554104731
    47893133       41298592       {1:10, 2:1, 4:9}
b2226820-b05b-11e5-a211-45b7aa88107c     system             hints
             1451634358178             18970740       18970740       {1:1}
7c7e0d80-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451532047960
    47584630       40254518       {1:10, 2:1, 4:9}
7d39a5d0-ad7f-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451319877805             3637           882
         {4:24}
c05d07c0-b1fe-11e5-a211-45b7aa88107c     system
sstable_activity             1451814341180             3329           852
         {4:23}
a227fdb0-add4-11e5-8bd4-45b7aa88107c     system             hints
             1451356446987             57070333       56869637       {2:3}
26a79a60-aeb5-11e5-8bd4-45b7aa88107c     system             hints
             1451452876550             57637242       57408618       {2:3}
def14230-af6c-11e5-b1dc-45b7aa88107c     system             hints
             1451531783635             57613272       57622817       {1:3}
ce8c20b0-afa6-11e5-a211-45b7aa88107c     system             local
             1451556666939             1167           696            {4:1}
75ace380-b1e5-11e5-a211-45b7aa88107c     system
sstable_activity             1451803478456             3171           820
         {4:23}
88d05020-af86-11e5-b1dc-45b7aa88107c     system             hints
             1451542806050             43921794       43937401       {2:3}
0a59ef20-b1f8-11e5-a211-45b7aa88107c     system             hints
             1451811458834             17972574       17713746       {1:1}
7ace0730-afa7-11e5-a211-45b7aa88107c     system             hints
             1451556955939             29776619       29776619       {1:2}
9e1ee8b0-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451532104379
    5935114        2171959        {4:9}
0580cea0-ae5c-11e5-8bd4-45b7aa88107c     system             hints
             1451414595722             57388662       57179650       {2:3}
378110c0-afa8-11e5-a211-45b7aa88107c     system             hints
             1451557272524             28568485       24640663       {1:1,
2:1}
ef231ab0-af5e-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451525797851             3739           973
         {4:25}
301fc6c0-ae25-11e5-8bd4-45b7aa88107c     system             hints
             1451391044908             56996239       56821882       {2:3}
5ceccc10-afa7-11e5-a211-45b7aa88107c     system             hints
             1451556905809             29776619       29776619       {1:2}
44713fe0-afa7-11e5-a211-45b7aa88107c     system             hints
             1451556864734             29776619       29776619       {1:2}
0b0786b0-afa7-11e5-a211-45b7aa88107c     system             hints
             1451556768411             29776619       29776619       {1:2}
5a667200-ae78-11e5-8bd4-45b7aa88107c     system             hints
             1451426764064             57497152       57334697       {2:3}
d4306ba0-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451531765594
    43985647       32914710       {1:7, 3:1, 4:9}
7596f1a0-ad9f-11e5-8bd4-45b7aa88107c     system             hints
             1451333608890             56854912       56608995       {1:3}
e31fc540-adee-11e5-8bd4-45b7aa88107c     system             hints
             1451367722900             57057877       56867011       {2:3}
52458b10-b1e3-11e5-a211-45b7aa88107c     system             hints
             1451802560065             18087954       17972574       {1:1}
278bf030-af13-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451493250739             3720           922
         {4:25}
76443290-af55-11e5-8bd4-45b7aa88107c     system             hints
             1451521729593             57500226       57294613       {2:3}
8065b450-af33-11e5-8bd4-45b7aa88107c     system             hints
             1451507143701             57557563       57359219       {2:3}
ae29e180-b0ce-11e5-a211-45b7aa88107c     system             hints
             1451683743640             18970740       18970740       {1:1}
d942a360-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451531774102
    8615536        4447310        {1:1, 3:1, 4:9}
6b039a10-b289-11e5-a211-45b7aa88107c     system             hints
             1451873898033             15967755       15796743       {1:1}
3afa2ba0-b14d-11e5-a211-45b7aa88107c     system
sstable_activity             1451738096474             3196           778
         {4:23}
3d6e34a0-afa7-11e5-a211-45b7aa88107c     system             hints
             1451556852970             29776619       29776619       {1:2}
b27e7970-af6c-11e5-b1dc-45b7aa88107c     system             local
             1451531709063             1215           668            {4:1}
78715730-ae62-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451417365539             3749           950
         {4:25}
46bd9b90-aed0-11e5-8bd4-45b7aa88107c     system             hints
             1451464526793             57589209       57452596       {2:3}
d9030250-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_SCO                1451531773685
    61554578       59278964       {1:76826, 2:11126, 3:604, 4:9}
c70ceb50-afcc-11e5-a211-45b7aa88107c     system             hints
             1451572975237             18970740       18970740       {1:1}
69f8a090-b106-11e5-a211-45b7aa88107c     system             hints
             1451707681048             18970740       18970740       {1:1}
7dd087b0-af79-11e5-b1dc-45b7aa88107c     system             hints
             1451537204139             43314155       43324175       {1:3}
94404a50-b166-11e5-a211-45b7aa88107c     system
sstable_activity             1451748983669             3364           926
         {4:23}
d99598d0-ad10-11e5-8bd4-45b7aa88107c     system             hints
             1451272358621             57162919       56981333       {2:3}
7fe03f20-b23d-11e5-a211-45b7aa88107c     system             hints
             1451841291282             17002773       16746317       {1:1}
e664d240-b1cc-11e5-a211-45b7aa88107c     system             hints
             1451792930148             18403648       18252171       {1:1}
04178e40-afa7-11e5-a211-45b7aa88107c     system             hints
             1451556756772             29776619       29776619       {1:2}
1a414650-ada6-11e5-8bd4-45b7aa88107c     system             hints
             1451336462133             56826857       56710683       {2:3}
ad9ef5f0-b026-11e5-a211-45b7aa88107c     system             hints
             1451611587279             18970740       18970740       {1:1}
60a4e320-ad17-11e5-8bd4-45b7aa88107c     system             hints
             1451275162194             57236552       57076664       {2:3}
917e6370-aeae-11e5-8bd4-45b7aa88107c     system             hints
             1451450049319             57539934       57384675       {2:3}
eb1440d0-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451531803997
    43867391       36299816       {1:8, 4:10}
a20bfc70-aee0-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451471551927             3833           945
         {4:25}
47cbe960-aef3-11e5-8bd4-45b7aa88107c     system             hints
             1451479560950             57443969       57255886       {2:3}
e2a84f00-af25-11e5-8bd4-45b7aa88107c     system             hints
             1451501295600             57449985       57298217       {2:3}
af1628a0-ade1-11e5-8bd4-45b7aa88107c     system             hints
             1451362052138             57001606       56839416       {2:3}
b0fea4a0-ae16-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451384818666             3645           872
         {3:1, 4:24}
15473980-aeae-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451449840920             3789           975
         {4:25}
e5bdc7c0-afa6-11e5-a211-45b7aa88107c     system             hints
             1451556705852             29776619       29776619       {1:2}
d7808360-afa0-11e5-84e3-45b7aa88107c     system
compactions_in_progress      1451554104982             461            42
          {2:2}
36a22720-ae63-11e5-8bd4-45b7aa88107c     system             hints
             1451417684626             57282895       57148294       {2:3}
99222cc0-ad54-11e5-8bd4-45b7aa88107c     system             hints
             1451301456268             57258179       57086571       {2:3}
a3480190-adb1-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451341416489             3683           925
         {4:24}
1198f0c0-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451531868620
    8937482        3569558        {2:1, 3:1, 4:8}
422d8470-b036-11e5-a211-45b7aa88107c     system
sstable_activity             1451618278967             3243           825
         {4:23}
12aee840-b218-11e5-a211-45b7aa88107c     system
sstable_activity             1451825216708             3176           819
         {4:23}
85af2f30-adb3-11e5-8bd4-45b7aa88107c     system             hints
             1451342225827             57028608       56811880       {2:3}
8b98e730-afa0-11e5-84e3-45b7aa88107c     system             hints
             1451553977634             88625803       44359037       {4:2,
5:1}
0c2b9aa0-ae02-11e5-8bd4-45b7aa88107c     system             hints
             1451375952202             56891360       56548629       {2:3}
ec9afff0-af00-11e5-8bd4-45b7aa88107c     system             hints
             1451485420911             57594351       57326414       {2:3}
98a23cb0-b199-11e5-a211-45b7aa88107c     system             hints
             1451770895355             18970740       18906200       {1:1}
8f90dde0-afb7-11e5-a211-45b7aa88107c     system
sstable_activity             1451563862718             3508           827
         {1:2, 2:8, 3:2, 4:19}
822b7da0-b14b-11e5-a211-45b7aa88107c     system             hints
             1451737356922             18970740       18970740       {1:1}
f5dc9cb0-ae54-11e5-8bd4-45b7aa88107c     system             hints
             1451411563003             57341080       57162518       {2:3}
83efd290-af8d-11e5-b1dc-45b7aa88107c     system             hints
             1451545804345             43996474       44000895       {2:3}
ba28f830-af80-11e5-b1dc-45b7aa88107c     system             hints
             1451540311859             43744401       43725750       {1:1,
2:2}
decb8560-afa6-11e5-a211-45b7aa88107c     system             hints
             1451556694198             29776619       29776619       {1:2}
8fb80860-ad5b-11e5-8bd4-45b7aa88107c     system             hints
             1451304446950             57324867       56998191       {2:3}
969514e0-afa7-11e5-a211-45b7aa88107c     system             hints
             1451557002542             29776619       29776619       {1:2}
bf7d00d0-afb5-11e5-a211-45b7aa88107c     system             hints
             1451563084125             18970740       18970740       {1:1}
113e2660-af48-11e5-8bd4-45b7aa88107c     system             hints
             1451515976646             57413867       57219902       {2:3}
32ee9f00-adf5-11e5-8bd4-45b7aa88107c     system             hints
             1451370433776             57034296       56618944       {2:3}
3c583550-afea-11e5-a211-45b7aa88107c     system
sstable_activity             1451585627429             3073           720
         {4:23}
58437cd0-af3a-11e5-8bd4-45b7aa88107c     system             hints
             1451510082845             57369449       57188682       {2:3}
871b8c30-b0ef-11e5-a211-45b7aa88107c     system             hints
             1451697851507             18970740       18970740       {1:1}
00522050-ae7f-11e5-8bd4-45b7aa88107c     system             hints
             1451429619413             57450075       57258800       {2:3}
14e3fd10-af6d-11e5-b1dc-45b7aa88107c     system
compactions_in_progress      1451531874145             472            42
          {2:2}
b3e23af0-b0e7-11e5-a211-45b7aa88107c     system
sstable_activity             1451694490655             3013           726
         {4:23}
aa5986d0-ad24-11e5-8bd4-45b7aa88107c     system             hints
             1451280869309             57181046       56983904       {2:3}
29697550-aebc-11e5-8bd4-45b7aa88107c     system             hints
             1451455887653             57617579       57451553       {2:3}
8201d200-ae8c-11e5-8bd4-45b7aa88107c     system             hints
             1451435420448             57558970       57354959       {2:3}
45bc2770-addb-11e5-8bd4-45b7aa88107c     system             hints
             1451359298407             56977263       56817508       {2:3}
ebd29a70-affe-11e5-a211-45b7aa88107c     system             hints
             1451594511767             18970740       18970740       {1:1}
9530ef00-ad4c-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451298013680             3580           885
         {4:24}
caa48640-afa6-11e5-a211-45b7aa88107c     system             local
             1451556660388             1013           677            {4:1}
acee31c0-aea0-11e5-8bd4-45b7aa88107c     system             hints
             1451444082396             57659639       57460800       {2:3}
aa2ad760-ae71-11e5-8bd4-45b7aa88107c     system             hints
             1451423891414             57468574       57242109       {2:3}
9e29cbd0-b036-11e5-a211-45b7aa88107c     system             hints
             1451618433293             18970740       18970740       {1:1}
df7b80d0-af6c-11e5-b1dc-45b7aa88107c     system
compactions_in_progress      1451531784541             542            172
         {1:2, 2:2}
48b11430-ad7d-11e5-8bd4-45b7aa88107c     system             hints
             1451318930675             56911864       56738503       {2:3}
6afb39b0-b199-11e5-a211-45b7aa88107c     system
sstable_activity             1451770818763             3039           788
         {4:23}
f77313f0-af40-11e5-8bd4-45b7aa88107c     system             hints
             1451512926895             57410690       57197897       {2:3}
38c86a20-adc1-11e5-8bd4-45b7aa88107c     system             hints
             1451348109762             56814910       56689825       {1:3}
c2406070-af0e-11e5-8bd4-45b7aa88107c     system             hints
             1451491362807             57391505       57316604       {2:3}
610527f0-b26f-11e5-a211-45b7aa88107c     system             hints
             1451862714351             16364535       16082982       {1:1}
23d18f10-ae49-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451406486145             3587           941
         {4:25}
76303090-ad40-11e5-8bd4-45b7aa88107c     system             hints
             1451292807705             57224902       57033897       {2:3}
787be5c0-ad76-11e5-8bd4-45b7aa88107c     system             hints
             1451316004380             56853811       56717047       {2:3}
f446f990-aef9-11e5-8bd4-45b7aa88107c     system             hints
             1451482427305             57466666       57373339       {2:3}
3ba17b30-ad47-11e5-8bd4-45b7aa88107c     system             hints
             1451295715939             57320671       57178557       {2:3}
85f76e20-afa8-11e5-a211-45b7aa88107c     system             hints
             1451557404162             24767833       20275337       {1:1,
2:1}
464b1550-af6d-11e5-b1dc-45b7aa88107c     system
compactions_in_progress      1451531957029             464            42
          {2:2}
d39fbc30-af9f-11e5-b1dc-45b7aa88107c     system
sstable_activity             1451553668979             3895           835
         {4:26}
acf90bb0-afa7-11e5-a211-45b7aa88107c     system             hints
             1451557040107             29214113       28457489       {1:1,
2:1}
d09de8e0-ad65-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451308850798             3685           899
         {4:24}
6e371d30-b27d-11e5-a211-45b7aa88107c     system             hints
             1451868749443             16082982       15967755       {1:1}
fca1eea0-ad83-11e5-8bd4-45b7aa88107c     system             hints
             1451321809546             56925900       56697130       {2:3}
31372130-b0f9-11e5-a211-45b7aa88107c     system             hints
             1451702002371             18970740       18970740       {1:1}
a0dd4c90-af45-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451514929113             3785           882
         {4:25}
a180b150-af6d-11e5-b1dc-45b7aa88107c     system
compactions_in_progress      1451532110053             446            0
         {2:2}
46239020-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1451531956770
    45648060       39245044       {1:8, 2:1, 3:1, 4:9}
76cf6680-afa0-11e5-84e3-45b7aa88107c     system             local
             1451553942760             1265           674            {4:1}
e4c92910-b11b-11e5-a211-45b7aa88107c     system             hints
             1451716906529             18970740       18970740       {1:1}
16e97320-b264-11e5-a211-45b7aa88107c     system
sstable_activity             1451857865554             3114           726
         {4:23}
fddd18f0-b100-11e5-a211-45b7aa88107c     system
sstable_activity             1451705352191             2959           716
         {4:23}
ea955a00-b173-11e5-a211-45b7aa88107c     system             hints
             1451754711968             18970740       18970740       {1:1}
52156430-adce-11e5-8bd4-45b7aa88107c     system             hints
             1451353735667             56989435       56824386       {2:3}
b9d73cf0-ae47-11e5-8bd4-45b7aa88107c     system             hints
             1451405878847             57223358       57041982       {2:3}
2ab91b60-afaf-11e5-a211-45b7aa88107c     system             schema_triggers
             1451560257558             134            60             {1:1,
3:1}
9712f8c0-ae39-11e5-8bd4-45b7aa88107c     system             hints
             1451399807564             57261044       57081560       {2:3}
86ec5560-adfb-11e5-8bd4-45b7aa88107c     system             hints
             1451373151670             56802348       56679018       {2:3}
4b9ffb20-b27d-11e5-a211-45b7aa88107c     system
sstable_activity             1451868691410             3037           791
         {4:23}
e0ed0e70-aef9-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451482394839             3749           941
         {4:25}
28e78950-afa7-11e5-a211-45b7aa88107c     system             hints
             1451556818533             29776619       29776619       {1:2}
47589f00-b231-11e5-a211-45b7aa88107c     system
sstable_activity             1451836042480             3212           785
         {4:23}
499c5d30-afaf-11e5-a211-45b7aa88107c     system             peers
             1451560309379             1810           556            {3:4}
2d335080-af6a-11e5-8bd4-45b7aa88107c     system             hints
             1451530626440             57406002       57261820       {2:3}
d46ed430-af6c-11e5-b1dc-45b7aa88107c     system
compactions_in_progress      1451531766003             608            268
         {1:2, 2:1}
31cbd580-af98-11e5-b1dc-45b7aa88107c     system             local
             1451550391000             905            741            {4:1}
95dec160-afa0-11e5-84e3-45b7aa88107c     system             hints
             1451553994870             44359037       44359037       {1:3}
4726d640-afe4-11e5-a211-45b7aa88107c     system             hints
             1451583068580             18970740       18970740       {1:1}
9ed33380-ae7b-11e5-8bd4-45b7aa88107c     system
sstable_activity             1451428167352             3638           906
         {4:25}
50fa9650-b1a8-11e5-a211-45b7aa88107c     system             hints
             1451777217589             18906200       18743329       {1:1}
b66e4420-b183-11e5-a211-45b7aa88107c     system             hints
             1451761496418             18970740       18970740       {1:1}
5a003890-ade8-11e5-8bd4-45b7aa88107c     system             hints
             1451364915865             57066041       56952696       {2:3}
451f4710-ad1e-11e5-8bd4-45b7aa88107c     system             hints
             1451278122497             57133634       56933373       {2:3}
b1e973b0-ad6f-11e5-8bd4-45b7aa88107c     system             hints
             1451313094251             56935662       56616319       {1:3}
0ce9d500-ae17-11e5-8bd4-45b7aa88107c     system             hints
             1451384972880             56788738       56649325       {2:3}
0fc24530-ae1e-11e5-8bd4-45b7aa88107c     system             hints
             1451387984131             56905778       56758301       {2:3}
31d54e10-b0da-11e5-a211-45b7aa88107c     system             hints
             1451688689009             18970740       18970740       {1:1}
cb322de0-b133-11e5-a211-45b7aa88107c     system
sstable_activity             1451727171518             3177           862
         {4:23}
4f987470-af1a-11e5-8bd4-45b7aa88107c     system             hints
             1451496324407             57567325       57405307       {2:3}
bbad5970-af2c-11e5-8bd4-45b7aa88107c     system             hints
             1451504236679             57519774       57367465       {2:3}
5e31ef40-aff4-11e5-a211-45b7aa88107c     system             hints
             1451589979188             18970740

On Mon, Jan 4, 2016 at 9:54 AM, Shuo Chen <chenatu2006@gmail.com> wrote:

> The attachment is the current system.log and config of the cassandra
> cluster. Over 8000 full gc happens in 3 days with 121 young gc.
>
> There are client operation in these days. Besides most columnfamily in the
> cluster are supercolumnfamily created by cassandra-cli. Most rows have
> average 30 sub-rows and each sub-row has 20 columns.
>
> You said that there are many compaction tasks. So how to check the source
> of this compactions? Thank you!
>
> On Fri, Jan 1, 2016 at 10:09 AM, Graham Sanderson <graham@vast.com> wrote:
>
>> If you are lucky that might mask the real issue, but I doubt it… that is
>> an insane number of compaction tasks and indicative of another problem. I
>> would check release notes of 2.0.6+, if I recall that was not a stable
>> version and may have had leaks.
>>
>> Aside from that, just FYI, if you use native_objects for memtables you
>> can run CMS fine even up to largish (20-30G) heap sizes without long GC
>> pauses (at least over months) . That said look for promotion failures of
>> 131074 DWORDS. Not sure what your GC logging options are, but what you
>> posted doesn’t say why a full GC happened which would be totally tunable,
>> but as I say I don’t think GC is actually your problem
>>
>>
>> On Dec 31, 2015, at 10:16 AM, Ipremyadav <ipremyadav@gmail.com> wrote:
>>
>> Simplest option is to use java 8 with G1 gc.
>>
>> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <chenatu2006@gmail.com> wrote:
>>
>> I have a cassandra 2.0.6 cluster with four nodes as backup database. The
>> only operation is posting data into db. Recently, the full gc of the nodes
>> increases apparently and blocks cluster operation.
>>
>> The load of each node is 10G. The heap is 8G each with default jvm memory
>> settings. The cpu is 24 cores. The settings of cassandra.yaml are almost
>> default.
>>
>> For debug, all the clients are disconnected, however the gc is still high.
>>
>> The GC is too often that it appears in every minute and blocks for nealy
>> 30s.
>>
>> How to debug this situation
>>
>> The following is the snippets of gc log:
>>
>> before full gc histogram
>>
>>
>> Total Free Space: 0
>> Max   Chunk Size: 0
>> Number of Blocks: 0
>> Tree      Height: 0
>> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max   Chunk Size: 0
>> Number of Blocks: 0
>> Tree      Height: 0
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max   Chunk Size: 0
>> Number of Blocks: 0
>> Tree      Height: 0
>> ----------------------------------------------
>>    1:      61030245     1952967840  java.util.concurrent.FutureTask
>>    2:      61031398     1464753552
>>  java.util.concurrent.Executors$RunnableAdapter
>>    3:      61030312     1464727488
>>  java.util.concurrent.LinkedBlockingQueue$Node
>>    4:      61030244     1464725856
>>  org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
>>    5:        214181       13386992  [B
>>    6:        202890        9738720  java.nio.HeapByteBuffer
>>    7:         37622        5994808  [C
>>    8:         42163        5722096  <constMethodKlass>
>>    9:         42163        5407152  <methodKlass>
>>   10:          4170        4648240  <constantPoolKlass>
>>   11:        100060        4002400
>>  org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
>>   12:          4170        2860816  <instanceKlassKlass>
>>   13:          3664        2702720  <constantPoolCacheKlass>
>>   14:          4817        2701576  [J
>>   15:          2575        1056600  <methodDataKlass>
>>   16:         37431         898344  java.lang.String
>>   17:          2585         627352  [Ljava.lang.Object;
>>   18:         17728         567296
>>  java.util.concurrent.ConcurrentHashMap$HashEntry
>>   19:         22111         530664  javax.management.ObjectName$Property
>>   20:          7595         463824  [S
>>   21:          4534         433560  java.lang.Class
>>   22:         11581         370592  java.util.HashMap$Entry
>>
>> ...
>> Total     289942930     8399196504
>> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
>> CMS: Large block 0x00000007f7da9588
>>
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 6335823
>> Max   Chunk Size: 6335823
>> Number of Blocks: 1
>> Av.  Block  Size: 6335823
>> Tree      Height: 1
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max   Chunk Size: 0
>> Number of Blocks: 0
>> Tree      Height: 0
>>
>>
>> It seems related to objects of Futuretask.
>>
>> --
>> *陈硕* *Shuo Chen*
>> chenatu2006@gmail.com
>> chenshuo@whaty.com
>>
>>
>>
>
>
> --
> *陈硕* *Shuo Chen*
> chenatu2006@gmail.com
> chenshuo@whaty.com
>



-- 
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com

Mime
View raw message