cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Wille <rwi...@fold3.com>
Subject Compaction not happening
Date Thu, 24 Sep 2015 19:59:28 GMT
I have some tables that have quite a bit of churn, and the deleted data doesn’t get compacted
out of them, in spite of gc_grace_seconds=0. I periodically get updates in bulk for some of
my tables. I write the new data to the tables, and then delete the old data (no updates, just
insertion with new primary keys, followed by deletion of the old records). On any given day,
I might add and delete 5 to 10 percent of the records. The amount of disk space that these
tables take up has historically been surprisingly constant. For many months, the space didn’t
vary by more than a few gigs. A couple of months ago, the tables started growing. They grew
to be about 50% bigger than they used to be, and they just kept growing. We decided to upgrade
our cluster (from 2.0.14 to 2.0.16), and right after the upgrade, the tables got compacted
down to their original size. The size then stayed pretty constant and I was feeling pretty
good about it. Unfortunately, a couple of weeks ago, they started growing again, and are now
about twice their original size. I’m using leveled compaction.

One thing I’ve noticed is that back when compaction was working great, whenever I’d start
writing to these tables, compactions would get triggered, and they would run for hours following
the bulk writing. Now, when I’m writing records, I see short little compactions that take
several seconds.

One other thing that may be relevant is that while I'm writing, max compactions pending can
get into the thousands, but drops to 0 as soon as I’m done writing. Seems quite strange
that Cassandra can chug through the pending compactions so quickly, while achieving so little.
Half the data in these tables can be compacted out, and yet compaction does almost nothing.

This seems really strange to me:

[cid:B0BC78C5-B998-40AF-BE4C-8EFC7F2110A4@iarchives.com] [cid:5B15BB63-1D1B-4C9F-AE80-C4E8B20CA6E2@iarchives.com]

Compactions pending shoots up when I’m done writing. Doesn’t make a lot of sense.

Any thoughts on how I can figure out what’s going on? Any idea what caused the tables to
be compacted following the upgrade? Any thoughts on why I used to have compactions that took
hours and actually did something, but now I get compactions that run really fast, but don’t
really do anything? Perhaps if I’m patient enough, the space will eventually get compacted
out, and yearning for the good-old days is just a waste of time. I can accept that, although
if that’s the case, I may need to buy more nodes.

Thanks in advance

Robert


Mime
View raw message