cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuo Chen <chenatu2...@gmail.com>
Subject Too many compactions, maybe keyspace system?
Date Fri, 08 Jan 2016 02:30:14 GMT
Hi,
I am using Cassandra 2.0.16 with 4 nodes and found too many compactions for
this cluster. This caused too much full gc and choked the system. I have
discussed the high gc in previous mails but didnot get the satisfied
answers.

To clarify the source of the compactions, I shutdown all the clients and
there is no reading and writing requests outside. Besides,
 hinted_handoff_enabled: false is set. The cluster is restarted yesterday
and have run for 1.5 days.

The heap is 8G and all the compaction settings are default settings. I
printed parts of gc histogram as follows:

 num     #instances         #bytes  class name
----------------------------------------------
   1:      67758530     2168272960  java.util.concurrent.FutureTask
   2:      67759745     1626233880
 java.util.concurrent.Executors$RunnableAdapter
   3:      67758576     1626205824
 java.util.concurrent.LinkedBlockingQueue$Node
   4:      67758529     1626204696
 org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
   5:         16935       72995576  [B
   6:        240534       11545632  java.nio.HeapByteBuffer
   7:         37416        5969800  [C
   8:         41447        5624856  <constMethodKlass>
   9:         41447        5315504  <methodKlass>
  10:        104850        5032800
 edu.stanford.ppl.concurrent.SnapTreeMap$Node
  11:          4110        4564144  <constantPoolKlass>
  12:        104781        3352992  org.apache.cassandra.db.Column
  13:          4110        2824016  <instanceKlassKlass>
  14:          4850        2689568  [J
  15:          3564        2634240  <constantPoolCacheKlass>
  16:          3016        1252544  <methodDataKlass>
  17:         37192         892608  java.lang.String
  18:         17641         564512
 java.util.concurrent.ConcurrentHashMap$HashEntry

As we can see in this list, objects of compaction tasks remain in the heap.

The nodetool compactionstats as follows:
pending tasks: 64637410
Active compaction remaining time :        n/a

Parts of nodetool compactionhistory:

Compaction History:
id                                       keyspace_name
 columnfamily_name            compacted_at              bytes_in
bytes_out      rows_merged
8e4f8830-b04f-11e5-a211-45b7aa88107c     system
sstable_activity             1451629144115             3342           915
         {4:23}
96a6fcb0-b04b-11e5-a211-45b7aa88107c     system             hints
             1451627440123             18970740       18970740       {1:1}
aefc3f10-b1b2-11e5-a211-45b7aa88107c     system
sstable_activity             1451781670273             3201           906
         {4:23}
1e76f1b0-b180-11e5-a211-45b7aa88107c     system
sstable_activity             1451759952971             3303           700
         {4:23}
e4d7d220-b518-11e5-ac1a-45b7aa88107c     system             hints
             1452155422786             7837400        7746243        {1:1}
f305ddd0-b52a-11e5-bce5-45b7aa88107c     system             hints
             1452163177517             5509472        0              {3:1}
5c5b15b0-b581-11e5-bce5-45b7aa88107c     system
sstable_activity             1452200290955             3387           866
         {4:23}
9cc8af70-b24a-11e5-a211-45b7aa88107c     system
sstable_activity             1451846923239             3134           773
         {4:23}
cf9e9400-b439-11e5-ac1a-45b7aa88107c     system             hints
             1452059609408             10524457       10490132       {3:1}
ea1b5140-b2e2-11e5-a211-45b7aa88107c     system
sstable_activity             1451912336468             3243           804
         {4:23}
a4b41910-b2b1-11e5-a211-45b7aa88107c     system             hints
             1451891174689             15426047       15236142       {1:3}
f1bb37a0-b204-11e5-a211-45b7aa88107c     system             hints
             1451817000986             17713746       17566349       {1:1}
fcb5bd60-b4d9-11e5-ac1a-45b7aa88107c     system
sstable_activity             1452128404534             3146           775
         {4:22}
dcd0c720-b0b4-11e5-a211-45b7aa88107c     system
sstable_activity             1451672654993             3189           797
         {4:23}
ba0f2d60-b50c-11e5-ac1a-45b7aa88107c     system
sstable_activity             1452150197046             3055           739
         {4:22}
a0681d00-b3ad-11e5-a211-45b7aa88107c     system
sstable_activity             1451999400656             3082           770
         {4:23}
ce6b4d90-b35c-11e5-a211-45b7aa88107c     system             hints
             1451964688617             13546056       13363986       {1:2}
a17d8f80-b24b-11e5-a211-45b7aa88107c     system             hints
             1451847360632             16746317       16600978       {1:1}
1549d4a0-b51c-11e5-bce5-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE             1452156792554
    46077372       41485236       {1:9, 2:4, 3:2, 4:5}
68e064a0-b439-11e5-ac1a-45b7aa88107c     system             local
             1452059437034             1255           680            {4:1}
38591c10-b22b-11e5-a211-45b7aa88107c     system             hints
             1451833440337             17351684       17002773       {1:1}
aba057f0-b1b4-11e5-a211-45b7aa88107c     system             hints
             1451782523631             18743329       18577220       {1:1}
1fe29f40-b568-11e5-bce5-45b7aa88107c     system
sstable_activity             1452189452084             3475           823
         {4:23}
f7e02540-b1bf-11e5-a211-45b7aa88107c     system             hints
             1451787376020             18577220       18403648       {1:1}
b3526be0-b3f9-11e5-a211-45b7aa88107c     system
sstable_activity             1452032074142             3344           795
         {1:2, 4:25}
2ffde460-b45b-11e5-ac1a-45b7aa88107c     system
sstable_activity             1452073944486             3318           835
         {4:22}
508ed070-b3fe-11e5-a211-45b7aa88107c     system             hints
             1452034055927             11448413       11309063       {1:2}
fdbf8750-b44c-11e5-ac1a-45b7aa88107c     system             hints
             1452067847237             10434789       10400837       {1:1}
9f5e38a0-b361-11e5-a211-45b7aa88107c     system
sstable_activity             1451966757162             3134           730
         {4:23}
621f05e0-b0ce-11e5-a211-45b7aa88107c     system
sstable_activity             1451683616062             3060           719
         {4:23}
5dde9ef0-b4b1-11e5-ac1a-45b7aa88107c     system             hints
             1452110958175             9239287        9010439        {1:1}
8f94db70-b51b-11e5-bce5-45b7aa88107c     system             local
             1452156568231             1201           688            {4:1}
4c414950-b32f-11e5-a211-45b7aa88107c     system             hints
             1451945142885             14105741       13937247       {1:2}
bd0eb510-b388-11e5-a211-45b7aa88107c     system             hints
             1451983557345             13062551       12823581       {1:2}
92707090-b09b-11e5-a211-45b7aa88107c     system
sstable_activity             1451661792793             3149           689
         {4:23}
aafc21e0-b497-11e5-ac1a-45b7aa88107c     system             hints
             1452099920638             9542212        9382531        {1:1}
55e055e0-b263-11e5-a211-45b7aa88107c     system             hints
             1451857541694             16600978       16364535       {1:1}
948bb3a0-b305-11e5-a211-45b7aa88107c     system             hints
             1451927225306             14648744       14472833       {1:2}
7946c2a0-b162-11e5-a211-45b7aa88107c     system             hints
             1451747220426             18970740       18970740       {1:1}
e5651290-b129-11e5-a211-45b7aa88107c     system             hints
             1451722920505             18970740       18970740       {1:1}
b2226820-b05b-11e5-a211-45b7aa88107c     system             hints
             1451634358178             18970740       18970740       {1:1}
11768640-b33f-11e5-a211-45b7aa88107c     system             hints
             1451951916196             13937247       13717550       {1:2}
c05d07c0-b1fe-11e5-a211-45b7aa88107c     system
sstable_activity             1451814341180             3329           852
         {4:23}
029d1530-b376-11e5-a211-45b7aa88107c     system             hints
             1451975513603             13204621       13062551       {1:2}
d1d10ac0-b441-11e5-ac1a-45b7aa88107c     system
sstable_activity             1452063049068             3395           816
         {1:3, 4:23}
0ec20100-b465-11e5-ac1a-45b7aa88107c     system             hints
             1452078183696             10217777       10090370       {1:1}
b25689d0-b474-11e5-ac1a-45b7aa88107c     system
sstable_activity             1452084900589             3221           776
         {4:22}
50e84480-b321-11e5-a211-45b7aa88107c     system             hints
             1451939137736             14276596       14105741       {1:2}
75ace380-b1e5-11e5-a211-45b7aa88107c     system
sstable_activity             1451803478456             3171           820
         {4:23}
d250c210-b441-11e5-ac1a-45b7aa88107c     system
schema_columnfamilies        1452063049905             185693         46547
         {1:1, 4:10}

As we can see, most compaction tasks come from keyspace system and
columnfamily hints. However hinted_handoff_enabled: false, so there may be
no new hints.

I checked the compaction settings of hints and sstable_activity.
hints:
compaction = {'enabled': 'false', 'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
sstable_activity:
compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}

The background compaction is disabled for cf hints.

Most of the other use keyspaces are created by cassandra-cli. In previous
mail discussion, somebody says these cfs have memory and performance
issues. But as we can see most compactions come from keyspace system.

So what is source of compactions? Is high gc really related to the high
compactions?

-- 
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com

Mime
View raw message