cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Viktor Jevdokimov <vjevdoki...@gmail.com>
Subject Re: how to increase compaction rate?
Date Tue, 13 Mar 2012 23:13:14 GMT
After loosing one node we had to repair, CFs was on leveled compaction.
For one CF each node had about 7GB of data.
Running a repair without primary range switch ended up some nodes exhausted
to about 60-100GB of 5MB sstables for that CF (a lot of files).
After switching back from leveled to tiered we ended up completely blocked
compactions on all nodes since this CF were compacting forever.
On one node a major compaction for that CF is CPU bound and may run with
unlimited compaction speed for 4-7 days at maximum 1MB/s rate, finally
compacting to 3GB of data (some data is deleted by TTL, some merged).

What we did to speedup this process to return all exhausted nodes into
normal state faster:
We have created a 6 temporary virtual single Cassandra nodes with 2 CPU
cores and 8GB RAM.
Stopped completely a compaction for CF on a production node.
Leveled sstables from this production node was divided into 6 ranges and
copied into 6 temporary empty nodes.
On each node we ran a major compaction to compact just 1/6 of data, about
10-14GB. It took 1-2 hours to compact them into 1GB of data.
Then all 6 sstables was copied into one of 6 nodes for a major compaction,
finally getting expected 3GB sstable.
Stopping production node, deleting files that was copied, returning
compacted (may need renaming) and node is back to normal.

Using separate nodes we saved original production nodes time not to compact
exhausted CF forever, blocking compactions for other CFs. With 6 separate
nodes we have compacted 2 productions nodes a day, so maybe it took the
same time, but production nodes were free for regular compactions for other
CFs.

After back to normal for our use case we stick to tiered compaction with a
major compaction nightly.
With our insertion/TTL deletion rates a leveled compaction is a nightmare,
even if amount of data is not very huge, just a few GBs/node.

2012/3/13 Thorsten von Eicken <tve@rightscale.com>

> On 3/12/2012 6:52 AM, Brandon Williams wrote:
> > On Mon, Mar 12, 2012 at 4:44 AM, aaron morton <aaron@thelastpickle.com>
> wrote:
> >> I don't understand why I
> >> don't get multiple concurrent compactions running, that's what would
> >> make the biggest performance difference.
> >>
> >> concurrent_compactors
> >> Controls how many concurrent compactions to run, by default it's the
> number
> >> of cores on the machine.
> I'm on a quad-core machine so not setting concurrent_compactors should
> not be a limiting factor...
> > With leveled compaction, I don't think you get any concurrency because
> > it has to compact an entire level, and it can't proceed to the next
> > level without completing the one before it.
> >
> > In short, if you want maximum throughput, stick with size tiered.
> I switched the CFs to tiered compaction and I still get no concurrency
> for the same CF. I now have two compactions running concurrently but
> always for different CFs. I've briefly seen a third for one of the small
> CFs, so it's willing to run more than two concurrently. Looks like I
> have to wait for a few days for all the compactions to complete. Talk
> about compaction hell!
>
> >
> > -Brandon
> >
>

Mime
View raw message