cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Wille <rwi...@fold3.com>
Subject Compaction no longer working properly
Date Fri, 24 Jul 2015 15:50:55 GMT
I have a database which has a fair amount of churn. When I need to update a data structure,
I create a new one, and when it is complete, I delete the old one. I have gc_grace_seconds=0,
so the space for the old data structures should be reclaimed on the next compaction. This
has been working fine for months. Unfortunately, compaction has stopped doing its job, and
about a week ago disk space started increasing steadily. Disk space usage has increased almost
50% in the last week.

I do see compaction tasks get kicked off, but they complete very quickly, generally in well
less than a minute. Compaction of the bigger tables used to take more than an hour. Admittedly,
its been quite a while since I’ve watched compactions run, and I probably haven’t watched
them since we last upgraded, so perhaps there’s been a change in how LeveledCompaction works.
Our last upgrade was well more than a week ago, so an upgrade isn’t responsible for this
change in behavior. We’re running 2.0.14 currently.

Looking at the logs, I see stuff like this:

INFO [CompactionExecutor:49180] 2015-07-24 09:31:45,960 CompactionTask.java (line 296) Compacted
2 sstables to [/var/lib/cassandra/data/fold31_browse/node/fold31_browse-node-jb-11110,]. 
117,066,921 bytes to 117,346,402 (~100% of original) in 33,766ms = 3.314288MB/s.  1,444,907
total partitions merged to 1,400,563.  Partition merge counts were {1:1356219, 2:44344, }

So, compaction seems to be doing something, just not very much. I’ve checked the process
that deletes stale data, and it is doing its job. Plenty of deletion going on, just no space
reclamation.

Any thoughts?

Thanks in advance

Robert


Mime
View raw message