cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shuo Chen (JIRA)" <>
Subject [jira] [Created] (CASSANDRA-11025) Too many compactions on certain node when too many empty tables are created
Date Mon, 18 Jan 2016 02:36:39 GMT
Shuo Chen created CASSANDRA-11025:

             Summary: Too many compactions on certain node when too many empty tables are
                 Key: CASSANDRA-11025
             Project: Cassandra
          Issue Type: Bug
          Components: Compaction
         Environment: 4 nodes with 24 core cpu and 32G ram, CentOS 6.5. Each node is configured
with 8G heap size.
            Reporter: Shuo Chen

I have configured 4 nodes of cassandra cluster version 2.0.6. Each node has about 10G load.
One of the node has too many pending compactions not too long and too many  full gc after
it restarts.

Here is part of gc histogram:
 num     #instances         #bytes  class name
   1:      67758530     2168272960  java.util.concurrent.FutureTask
   2:      67759745     1626233880  java.util.concurrent.Executors$RunnableAdapter
   3:      67758576     1626205824  java.util.concurrent.LinkedBlockingQueue$Node
   4:      67758529     1626204696  org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
   5:         16935       72995576  [B
   6:        240534       11545632  java.nio.HeapByteBuffer
   7:         37416        5969800  [C
   8:         41447        5624856  <constMethodKlass>
   9:         41447        5315504  <methodKlass>
  10:        104850        5032800  edu.stanford.ppl.concurrent.SnapTreeMap$Node
  11:          4110        4564144  <constantPoolKlass>
  12:        104781        3352992  org.apache.cassandra.db.Column
  13:          4110        2824016  <instanceKlassKlass>

Here is the nodetool stats:

[cassandra@whaty181 apache-cassandra-2.0.16]$ bin/nodetool compactionstats
pending tasks: 64642341
Active compaction remaining time :        n/a

However in system.log, there are not too much log of compaction. I used inotify to monitor
the data directory events. There are few events when pending tasks accumulate.

I used jmap to dump the heap and analyzed the java.util.concurrent.FutureTask. It contains
many task of CompactionExecutor. I checked the cf of CompactionExecutor. However most of the
cfs are never inserted any data since created.

I have created 6 keyspaces and 5 keyspaces never inserted any data. In these 5 keyspaces,
2 keyspaces contain 42 cfs each and 3 keyspaces contains 6 cfs in total. So there are about
90 empty  cfs.

After I dropped these 5 keyspaces and restarted this node, the compaction status of this node
is normal again. So I suggests there is many some bug concerning compaction on empty columnfamily.

This message was sent by Atlassian JIRA

View raw message