cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Lohfink <clohf...@blackbirdit.com>
Subject Re: high pending compactions
Date Tue, 15 Jul 2014 23:40:12 GMT
Number of compactions can spike for perfectly reasonable scenarios (ie after a repair or bootstrap)
so no matter what you set it to there will likely be periodic false alarms.  

I would however put a number "15" on the limit of any thread pool backlog as matter of principle
until woken up too many times in middle of night.  concurrent compactors will likely be to
low (depending on number of cores).

---
Chris Lohfink

On Jul 14, 2014, at 7:31 PM, Greg Bone <gbone24@gmail.com> wrote:

> I'm looking into creation of monitoring thresholds for cassandra to report on its health.
Does it make sense to set an alert threshold on compaction stats? If so, would setting it
to a value equal to or greater than concurrent compactions make sense?
> 
> Thanks,
> Greg
> 
> 
> 
> 
> On Mon, Jun 9, 2014 at 2:14 PM, S C <asf11@outlook.com> wrote:
> Thank you all for quick responses.
> From: clohfink@blackbirdit.com
> Subject: Re: high pending compactions
> Date: Mon, 9 Jun 2014 14:11:36 -0500
> To: user@cassandra.apache.org
> 
> Bean: org.apache.cassandra.db.CompactionManager
> 
> also nodetool compactionstats gives you how many are in the queue + estimate of how many
will be needed.
> 
> in 1.1 you will OOM far before you hit the limit,.  In theory though, the compaction
executor is a little special cased and will actually throw an exception (normally it will
block)
> 
> Chris
> 
> On Jun 9, 2014, at 7:49 AM, S C <asf11@outlook.com> wrote:
> 
> Thank you all for valuable suggestions. Couple more questions,
> 
> How to check the compaction queue? MBean/C* system log ?
> What happens if the queue is full?
> 
> From: colinkuo.tw@gmail.com
> Date: Mon, 9 Jun 2014 18:53:41 +0800
> Subject: Re: high pending compactions
> To: user@cassandra.apache.org
> 
> As Jake suggested, you could firstly increase "compaction_throughput_mb_per_sec" and
"concurrent_compactions" to suitable values if system resource is allowed. From my understanding,
major compaction will internally acquire lock before running compaction. In your case, there
might be a major compaction blocking the pending following compaction tasks. You could check
the result of "nodetool compactionstats" and C* system log for double confirm.
> 
> If the running compaction is compacting wide row for a long time, you could try to tune
"in_memory_compaction_limit_in_mb" value.
> 
> Thanks,
> 
> 
> 
> On Sun, Jun 8, 2014 at 11:27 PM, S C <asf11@outlook.com> wrote:
> I am using Cassandra 1.1 (sorry bit old) and I am seeing high pending compaction count.
"pending tasks: 67" while active compaction tasks are not more than 5. I have a 24CPU machine.
Shouldn't I be seeing more compactions? Is this a pattern of high writes and compactions backing
up? How can I improve this? Here are my thoughts.
> 
> Increase memtable_total_space_in_mb
> Increase compaction_throughput_mb_per_sec
> Increase concurrent_compactions
> 
> Sorry if this was discussed already. Any pointers is much appreciated. 
> 
> Thanks,
> Kumar
> 
> 


Mime
View raw message