cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-2901) Allow taking advantage of multiple cores while compacting a single CF
Date Sun, 17 Jul 2011 20:57:00 GMT


Jonathan Ellis commented on CASSANDRA-2901:

That's an interesting idea, but the more I think about it the less convinced I am that it's
an easy win.

First of all, the premise that compaction is GC-intensive should be qualified: it can help
cause young-gen compactions, but almost none of it will ever be promoted to old gen, which
is what most people worry about.  Small rows are compacted quickly enough to not be promoted,
and large rows compact column-at-a-time which will also not live long enough to be promoted.
 (If you are seeing "medium size" rows get tenured, then consider reduction in_memory_compaction_limit_in_mb.)

Second, it's harder than it looks to actually push compaction out to another process, because
you have basically three choices:
- use Runtime.exec or ProcessBuilder
- use JNA and vfork
- run a separate, always-on "compaction daemon" and communicate with it over RMI or other

The first of these is implemented using fork on Linux, which can cause spurious OOMs when
running in an environment with overcommit disabled (which is generally accepted as best practice
in a server environment). Overcommit aside, copying even just the page table for a largish
heap is expensive: 

vfork allows avoiding copying the parent process's page table, but is obviously not completely
portable so we'd have to keep in-process compaction around as a fallback option.

Neither of these makes it easy to communicate back to the parent Cassandra process what cache
rows should be invalidated (CASSANDRA-2305). This may be something we can live with (we did
for years), but it's a regression nevertheless.

The compaction daemon approach avoids the above problems but adds substantial complexity to

tl;dr: you're welcome to experiment with it but I don't think it's at all clear yet that the
cost/benefit is there.

> Allow taking advantage of multiple cores while compacting a single CF
> ---------------------------------------------------------------------
>                 Key: CASSANDRA-2901
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Jonathan Ellis
>            Priority: Minor
> Moved from CASSANDRA-1876:
> There are five stages: read, deserialize, merge, serialize, and write. We probably want
to continue doing read+deserialize and serialize+write together, or you waste a lot copying
to/from buffers.
> So, what I would suggest is: one thread per input sstable doing read + deserialize (a
row at a time). One thread merging corresponding rows from each input sstable. One thread
doing serialize + writing the output. This should give us between 2x and 3x speedup (depending
how much doing the merge on another thread than write saves us).
> This will require roughly 2x the memory, to allow the reader threads to work ahead of
the merge stage. (I.e. for each input sstable you will have up to one row in a queue waiting
to be merged, and the reader thread working on the next.) Seems quite reasonable on that front.
> Multithreaded compaction should be either on or off. It doesn't make sense to try to
do things halfway (by doing the reads with a
> threadpool whose size you can grow/shrink, for instance): we still have compaction threads
tuned to low priority, by default, so the impact on the rest of the system won't be very different.
Nor do we expect to have so many input sstables that we lose a lot in context switching between
reader threads. (If this is a concern, we already have a tunable to limit the number of sstables
merged at a time in a single CF.)
> IMO it's acceptable to punt completely on rows that are larger than memory, and fall
back to the old non-parallel code there. I don't see any sane way to parallelize large-row

This message is automatically generated by JIRA.
For more information on JIRA, see:


View raw message