incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Derek Bromenshenkel <derek.bromenshen...@gmail.com>
Subject Re: How to determine compaction bottlenecks
Date Thu, 29 Nov 2012 05:50:16 GMT
aaron morton <aaron <at> thelastpickle.com> writes:

> 
> 
> 
> I've been playing around with trying to figure out what is making 
compactions run so slow.Is this regular compaction or table upgrades ? 
> I *think* upgrade tables is single threaded. 
> Do you have some compaction logs lines that say "Compacted to…"? It's handy to 
see the throughput and the number of keys compacted.
> 
> snapshot_before_compaction: falsein_memory_compaction_limit_in_mb: 
256multithreaded_compaction: truecompaction_throughput_mb_per_sec: 
128compaction_preheat_key_cache: true
> What setting for concurrent_compactors ? 
> I would also check the logs for GC issues. 
> 
> Cheers
> 
> 
> 
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
> 
>  <at> aaronmorton
> http://www.thelastpickle.com
> 
> 
> 
> 
> On 28/11/2012, at 4:23 AM, Derek Bromenshenkel <derek.bromenshenkel <at> 
gmail.com> wrote:
> Setup: C* 1.1.6, 6 node (Linux, 64GB RAM, 16 Core CPU, 2x512 SSD), RF=3, 
1.65TB total usedBackground: Client app is off - no reads/writes happening. 
Doing some cluster maintenance requiring node repairs and upgradesstables.I've 
been playing around with trying to figure out what is making compactions run so 
slow.  Watching syslogs, it seems to average 3-4MB/s.  That just seems so slow 
for this set up and the fact there is zero external load on the cluster.  As far 
as I can tell:1. Not I/O bound according to iostat data 2. CPU seems to be 
idiling also3. From my understanding, I am using all the correct compaction 
settings for this setup: Here are those below:snapshot_before_compaction: 
falsein_memory_compaction_limit_in_mb: 256multithreaded_compaction: 
truecompaction_throughput_mb_per_sec: 128compaction_preheat_key_cache: trueSome 
other thoughts:- I have turned on DEBUG logging for the Throttle class and 
played with the live compaction_throughput_mb_per_sec setting.  I can see it 
performing the throttling if I set the value low (say 4), but anything over 8 it 
is apparently running wide open. [Side note: Although the math for the Throttle 
class adds up, over all the throttling seems to be very very conservative.]- I 
accidently turned on DEBUG for the entire ...compaction.* package and that 
unintentionally created A LOT of I/O from the ParallelCompactionIterable class, 
and the disk/OS handled that just fine.Perhaps I just don't fully grasp what is 
going on or have the correct expectations.  I am OK with things being slow if 
the hardware is working hard, but that does not seem to be the case.Anyone have 
some insight?Thanks
> 
> 
> 
> 
> 
> 

Hi Aaron,

Thank you for taking the time and responding.  I'll try to answer your 
questions.

- reg vs upgrade: Seeing the same speed on regular compaction and upgrades.  
True that most of the frustration comes from the upgrades since there is so much 
work to be done.
- GC: looked fine. I've seen pressure before, but only when under very heavy 
client app load.
- concurrent_compactors: not set, so should default to #cores [32; 16 phys * 2 
hyperthread], and I did see 32 CompactionExecuter (I think) threads via JMX.
- examples: yes I have a lot of examples. here are some

Leveled
 INFO [CompactionExecutor:1033] 2012-11-26 01:38:56,800 CompactionTask.java 
(line 221) Compacted to [lcs1].  35,058,450 to 33,408,896 (~95% of original) 
bytes for 127,771 keys at 4.371735MB/s.  Time: 7,288ms.
 INFO [CompactionExecutor:2015] 2012-11-26 03:12:43,800 CompactionTask.java 
(line 221) Compacted to [lcs2].  37,029,581 to 36,747,459 (~99% of original) 
bytes for 135,471 keys at 3.748541MB/s.  Time: 9,349ms.

Size Tiered
 INFO [CompactionExecutor:6242] 2012-11-26 10:46:24,030 CompactionTask.java 
(line 221) Compacted to [abc].  12,804,781,130 to 5,575,340,207 (~43% of 
original) bytes for 84,723,404 keys at 1.382544MB/s.  Time: 3,845,851ms.
 INFO [CompactionExecutor:288] 2012-11-26 00:42:58,629 CompactionTask.java (line 
221) Compacted to [def].  116,347,764 to 58,354,237 (~50% of original) bytes for 
2,511,375 keys at 0.655612MB/s.  Time: 84,884ms.
 INFO [CompactionExecutor:5113] 2012-11-26 08:33:12,885 CompactionTask.java 
(line 221) Compacted to [ghi].  560,682,371 to 294,965,985 (~52% of original) 
bytes for 220 keys at 3.172669MB/s.  Time: 88,664ms.
 INFO [CompactionExecutor:6124] 2012-11-26 09:36:52,141 CompactionTask.java 
(line 221) Compacted to [jkl].  418,807,103 to 234,394,618 (~55% of original) 
bytes for 3,130,751 keys at 2.807220MB/s.  Time: 79,629ms.

Also, upon reading the messages here/JIRA/etc I decided to disable 
multithreaded_compaction late yesterday.  That helped to the tune of 3-5x 
improvement. Why multi is so much slower I'm willing to digress for now.  
However, I'm still interested in understanding why, under zero load in an 
unthrottled state, the compaction process does not consume at least one full CPU 
core and/or max out the disk I/O.

Thanks again,
Derek


Mime
View raw message