incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: High CPU usage during repair
Date Sun, 10 Feb 2013 20:04:36 GMT
> During repair I see high CPU consumption, 
Repair reads the data and computes a hash, this is a CPU intensive operation.
Is the CPU over loaded or is just under load?

> I run Cassandra  version 1.0.11, on 3 node setup on EC2 instances.
What machine size?

> there are compactions waiting.
That's normally ok. How many are waiting?

> I thought of adding a call to my repair script, before repair starts to do:
> nodetool setcompactionthroughput 0
> and then when repair finishes call
> nodetool setcompactionthroughput 16
That will remove throttling on compaction and the validation compaction used for the repair.
Which may in turn add additional IO load, CPU load and GC pressure. You probably do not want
to do this. 

Try reducing the compaction throughput to say 12 normally and see the effect.

Cheers


-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 11/02/2013, at 1:01 AM, Tamar Fraenkel <tamar@tok-media.com> wrote:

> Hi!
> I run repair weekly, using a scheduled cron job.
> During repair I see high CPU consumption, and messages in the log file
> "INFO [ScheduledTasks:1] 2013-02-10 11:48:06,396 GCInspector.java (line 122) GC for ParNew:
208 ms for 1 collections, 1704786200 used; max is 3894411264"
> From time to time, there are also messages of the form
> "INFO [ScheduledTasks:1] 2012-12-04 13:34:52,406 MessagingService.java (line 607) 1 READ
messages dropped in last 5000ms"
> 
> Using opscenter, jmx and nodetool compactionstats I can see that during the time the
CPU consumption is high, there are compactions waiting.
> 
> I run Cassandra  version 1.0.11, on 3 node setup on EC2 instances.
> I have the default settings:
> compaction_throughput_mb_per_sec: 16
> in_memory_compaction_limit_in_mb: 64
> multithreaded_compaction: false
> compaction_preheat_key_cache: true
> 
> I am thinking on the following solution, and wanted to ask if I am on the right track:
> I thought of adding a call to my repair script, before repair starts to do:
> nodetool setcompactionthroughput 0
> and then when repair finishes call
> nodetool setcompactionthroughput 16
> 
> Is this a right solution?
> Thanks,
> Tamar
> 
> Tamar Fraenkel 
> Senior Software Engineer, TOK Media 
> 
> <tokLogo.png>
> 
> tamar@tok-media.com
> Tel:   +972 2 6409736 
> Mob:  +972 54 8356490 
> Fax:   +972 2 5612956 
> 
> 


Mime
View raw message