cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl Yeksigian (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CASSANDRA-11025) Too many compactions on certain node when too many empty tables are created
Date Tue, 19 Jan 2016 18:11:39 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-11025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Carl Yeksigian resolved CASSANDRA-11025.
----------------------------------------
       Resolution: Duplicate
    Fix Version/s: 2.0.17

This appears to be a dupe of CASSANDRA-9662, which was fixed in 2.0.17.

Note that 2.0 is EOL, so no new patches or releases will be made. If you try with 2.1 and
find the same issue, please reopen.

> Too many compactions on certain node when too many empty tables are created
> ---------------------------------------------------------------------------
>
>                 Key: CASSANDRA-11025
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11025
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: 4 nodes with 24 core cpu and 32G ram, CentOS 6.5. Each node is configured
with 8G heap size.
>            Reporter: Shuo Chen
>             Fix For: 2.0.17
>
>
> I have configured 4 nodes of cassandra cluster version 2.0.16. Each node has about 10G
load. One of the node has too many pending compactions not too long and too many  full gc
after it restarts.
> Here is part of gc histogram:
>  num     #instances         #bytes  class name
> ----------------------------------------------
>    1:      67758530     2168272960  java.util.concurrent.FutureTask
>    2:      67759745     1626233880  java.util.concurrent.Executors$RunnableAdapter
>    3:      67758576     1626205824  java.util.concurrent.LinkedBlockingQueue$Node
>    4:      67758529     1626204696  org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
>    5:         16935       72995576  [B
>    6:        240534       11545632  java.nio.HeapByteBuffer
>    7:         37416        5969800  [C
>    8:         41447        5624856  <constMethodKlass>
>    9:         41447        5315504  <methodKlass>
>   10:        104850        5032800  edu.stanford.ppl.concurrent.SnapTreeMap$Node
>   11:          4110        4564144  <constantPoolKlass>
>   12:        104781        3352992  org.apache.cassandra.db.Column
>   13:          4110        2824016  <instanceKlassKlass>
> Here is the nodetool stats:
> [cassandra@whaty181 apache-cassandra-2.0.16]$ bin/nodetool compactionstats
> pending tasks: 64642341
> Active compaction remaining time :        n/a
> However in system.log, there are not too much log of compaction. I used inotify to monitor
the data directory events. There are few events when pending tasks accumulate.
> I used jmap to dump the heap and analyzed the java.util.concurrent.FutureTask. It contains
many task of CompactionExecutor. I checked the cf of CompactionExecutor. However most of the
cfs are never inserted any data since created.
> I have created 6 keyspaces and 5 keyspaces never inserted any data. In these 5 keyspaces,
2 keyspaces contain 42 cfs each and 3 keyspaces contains 6 cfs in total. So there are about
90 empty  cfs.
> All of these cfs are super column families created using casandra-cli
> After I dropped these 5 keyspaces and restarted this node, the compaction status of this
node is normal again. So I suggests there is many some bug concerning compaction on empty
columnfamily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message