cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra pending compaction tasks keeps increasing
Date Mon, 21 Jan 2013 08:21:29 GMT
The main guarantee LCS gives you is that most reads will only touch 1 row http://www.datastax.com/dev/blog/when-to-use-leveled-compaction

If compaction is falling behind this may not hold.

nodetool cfhistograms tells you how many SSTables were read from for reads.  It's a recent
histogram that resets each time you read from it. 

Also, parallel levelled compaction in 1.2 http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 20/01/2013, at 7:49 AM, Jim Cistaro <jcistaro@netflix.com> wrote:

> 1) In addition to iostat, dstat is a good tool to see wht kind of disck throuput your
are getting.  That would be one thing to monitor.
> 2) For LCS, we also see pending compactions skyrocket.  During load, LCS will create
a lot of small sstables which will queue up for compaction.
> 3) For us the biggest concern is not how high the pending count gets, but how often it
gets back down near zero.  If your load is something you can do in segments or pause, then
you can see how fast the cluster recovers on the compactions.
> 4) One thing which we tune per cluster is the size of the files.  Increasing this from
5MB can sometimes improve things.  But I forget if we have ever changed this after starting
data load.
> 
> Is your cluster receiving read traffic during this data migration? If so, I would say
that read latency is your best measure.  If the high number of SSTables waiting to compact
is not hurting your reads, then you are probably ok.  Since you are on SSD, there is a good
chance the compactions are not hurting you.  As for compactionthroughput, we set ours high
for SSD.  You usually wont use it all because the compactions are usually single threaded.
 Dstat will help you measure this.
> 
> I hope this helps,
> jc
> 
> From: Wei Zhu <wz1975@yahoo.com>
> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>, Wei Zhu <wz1975@yahoo.com>
> Date: Friday, January 18, 2013 12:10 PM
> To: Cassandr usergroup <user@cassandra.apache.org>
> Subject: Cassandra pending compaction tasks keeps increasing
> 
> Hi,
> When I run nodetool compactionstats
> 
> I see the number of pending tasks keep going up steadily. 
> 
> I tried to increase the  compactionthroughput, by using
> 
> nodetool setcompactionthroughput
> 
> I even tried the extreme to set it to 0 to disable the throttling. 
> 
> I checked iostats and we have SSD for data, the disk util is less than 5% which means
it's not I/O bound, CPU is also less than 10%
> 
> We are using levelcompaction and in the process of migrating data. We have 4500 writes
per second and very few reads. We have about 70G data now and will grow to 150G when the migration
finishes. We only have one CF and right now the number of  SSTable is around 15000, write
latency is still under 0.1ms. 
> 
> Anything needs to be concerned? Or anything I can do to reduce the number of pending
compaction?
> 
> Thanks.
> -Wei
> 
> 


Mime
View raw message