Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of ashaw@wetafx.co.nz designates
 110.232.144.26 as permitted sender)
Message-ID: <4DF93037.80603@wetafx.co.nz>
Date: Thu, 16 Jun 2011 10:20:39 +1200
From: Aurynn Shaw <ashaw@wetafx.co.nz>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.17) Gecko/20110430 Lightning/1.0b2 Thunderbird/3.1.10
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Slowdowns during repair
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hey all;

So, we have Cassandra running on a 5-server ring, with a RF of 3, and 
we're regularly seeing major slowdowns in read & write performance while 
running nodetool repair, as well as the occasional Cassandra crash 
during the repair window - slowdowns past 10 seconds to perform a single 
write.

The repair cycle runs nightly on a different server, so each server has 
it run once a week.

We're running 0.7.0 currently, and we'll be upgrading to 0.7.6 shortly.

System load on the Cassandra servers is never more than 10% CPU and 
utterly minimal IO usage, so I wouldn't think we'd be seeing issues 
quite like this.

What sort of knobs should I be looking at tuning to reduce the impact 
that nodetool repair has on Cassandra? What questions should I be asking 
as to why Cassandra slows down to the level that it does, and what I 
should be optimizing?

Additionally, what should I be looking for in the logs when this is 
happening? There's a lot in the logs, but I'm not sure what to look for.

Cassadra is, in this instance, backing a system that supports around a 
million requests a day, so not terribly heavy traffic.

Thanks,

Aurynn