cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: Capacity problem with a lot of writes?
Date Fri, 26 Nov 2010 00:09:39 GMT
>> When you say that it grows constantly, does that mean up to 30 or even
>> farther?
> My total data size is 2TB

For the entire cluster? (I realized I was being ambiguous, I meant per node.)

> Actually, I never see the count stable.  When it reached 30 I thinked
> "I am reaching the default upper limit for a compaction, something
> went wrong" and I went back to 1GB memtables (also, I saw bigger read
> latencies).

Assuming 2 TB is for the entire cluster and you have 8 nodes, that's
250 GB per node. So yeah, that'll take a reasonable time to compact
when larger or major compactions take place.

> Well, I think you are right: I am CPU bounded on compaction, because I
> see during compactions a single jvm thread which is almost all the
> time in running state and the disk is not used beyond 50%.

Sounds like it.

> I think that a partial solution would help: if the compaction
> compacted to 'n' diferents new sstables (not one), the implementation
> would be easier. I mean, the compaction would compact, for instance,
> 10 sstables to 2 (being 2 the level of paralelism). In this way, the
> sstables count would remain eventually stable (although higher). What
> do you think?

Yes, that's been my thinking. Even for a single huge column family, it
doesn't really matter that individual large compactions take longer as
long as smaller compactions keep happening concurrently.

It's worth noting though that merely supporting concurrent compactions
in the sense of spawning more than one thread to do it is only a
partial solution; with sufficiently large column families you end up
having to keep several (not just a couple) compactions going in order
to ensure that large and medium compactions are run while at the same
time allowing the smallest compactions to take place. For one thing
you probably don't want to commit too many cores to compactions, and
in addition you can run into issues with I/O becoming seek bound if
you have too many processes trying to stream data concurrently.

I think the optimal solution for this would involve something along
the lines of having a fixed configurable compaction machine
concurrency (number of threads), and then have those threads
interleave an arbitrary number of compactions (do 1 gig, switch to the
other compaction, do 1 gig, etc). That way you have direct control
over actual CPU and I/O concurrency, and can independently make sure
that compactions happen in proper prioritized order, constantly
keeping sstable counts within appropriate bounds.

(For comparison the problem is very similar to PostgreSQL and
auto-vacuuming of databases containing tables with extreme differences
in size. Allowing multiple vacuum processes helps solve this, but is
has the equivalent scalability issues if you were to have a
particularly problematic distribution of table sizes and workload.)

/ Peter Schuller

View raw message