cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aurynn Shaw <as...@wetafx.co.nz>
Subject Slowdowns during repair
Date Wed, 15 Jun 2011 22:20:39 GMT
Hey all;

So, we have Cassandra running on a 5-server ring, with a RF of 3, and 
we're regularly seeing major slowdowns in read & write performance while 
running nodetool repair, as well as the occasional Cassandra crash 
during the repair window - slowdowns past 10 seconds to perform a single 
write.

The repair cycle runs nightly on a different server, so each server has 
it run once a week.

We're running 0.7.0 currently, and we'll be upgrading to 0.7.6 shortly.

System load on the Cassandra servers is never more than 10% CPU and 
utterly minimal IO usage, so I wouldn't think we'd be seeing issues 
quite like this.

What sort of knobs should I be looking at tuning to reduce the impact 
that nodetool repair has on Cassandra? What questions should I be asking 
as to why Cassandra slows down to the level that it does, and what I 
should be optimizing?

Additionally, what should I be looking for in the logs when this is 
happening? There's a lot in the logs, but I'm not sure what to look for.

Cassadra is, in this instance, backing a system that supports around a 
million requests a day, so not terribly heavy traffic.

Thanks,

Aurynn

Mime
View raw message