cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jeff.ji...@crowdstrike.com>
Subject Re: large number of pending compactions, sstables steadily increasing
Date Wed, 17 Aug 2016 18:49:14 GMT
What compaction strategy? Looks like leveled – is that what you expect? 

 

Any exceptions in the logs? 

 

Are you throttling compaction?

 

SSD or spinning disks?

 

How many cores?

 

How many concurrent compactors? 

 

 

 

From: Ezra Stuetzel <ezra.stuetzel@riskiq.net>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wednesday, August 17, 2016 at 11:39 AM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: large number of pending compactions, sstables steadily increasing

 

I have one node in my cluster 2.2.7 (just upgraded from 2.2.6 hoping to fix issue) which seems
to be stuck in a weird state -- with a large number of pending compactions and sstables. The
node is compacting about 500gb/day, number of pending compactions is going up at about 50/day.
It is at about 2300 pending compactions now. I have tried increasing number of compaction
threads and the compaction throughput, which doesn't seem to help eliminate the many pending
compactions.   

 

I have tried running 'nodetool cleanup' and 'nodetool compact'. The latter has fixed the issue
in the past, but most recently I was getting OOM errors, probably due to the large number
of sstables. I upgraded to 2.2.7 and am no longer getting OOM errors, but also it does not
resolve the issue. I do see this message in the logs:

 

INFO  [RMI TCP Connection(611)-10.9.2.218] 2016-08-17 01:50:01,985 CompactionManager.java:610
- Cannot perform a full major compaction as repaired and unrepaired sstables cannot be compacted
together. These two set of sstables will be compacted separately.

Below are the 'nodetool tablestats' comparing a normal and the problematic node. You can see
problematic node has many many more sstables, and they are all in level 1. What is the best
way to fix this? Can I just delete those sstables somehow then run a repair?

Normal node

keyspace: mykeyspace

    Read Count: 0

    Read Latency: NaN ms.

    Write Count: 31905656

    Write Latency: 0.051713177939359714 ms.

    Pending Flushes: 0

        Table: mytable

        SSTable count: 1908

        SSTables in each level: [11/4, 20/10, 213/100, 1356/1000, 306, 0, 0, 0, 0]

        Space used (live): 301894591442

        Space used (total): 301894591442

 

 

Problematic node

Keyspace: mykeyspace

    Read Count: 0

    Read Latency: NaN ms.

    Write Count: 30520190

    Write Latency: 0.05171286705620116 ms.

    Pending Flushes: 0

        Table: mytable

        SSTable count: 14105

        SSTables in each level: [13039/4, 21/10, 206/100, 831, 0, 0, 0, 0, 0]

        Space used (live): 561143255289

        Space used (total): 561143255289

Thanks,

Ezra


Mime
View raw message