incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: puzzled why my cluster is slowing down
Date Mon, 07 Jan 2013 20:47:27 GMT
Can you slice up the "slows down" part a little more? 
Are you saying you are getting 4500 u-sec write latency ? 

Are you using secondary indexes?
What sort of read queries are slowing down? 
What does the schema look like ? 

If the simple checks like CPU, iostat and GC logging in the cassandra log do not show anything
the next thing to look at is work load. Look for things like wide gets on wide rows, long
lived rows with many deletes. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 8/01/2013, at 3:39 AM, Brian Tarbox <tarbox@cabotresearch.com> wrote:

> I have a 4 node cluster with lots JVM memory and lots of system memory that slows down
when I'm doing lots of writes.
> 
> Running DataStax charts I see my read and write latency rise from 50-100 u-secs to 1500-4500
u-secs.  This is across a 12 hour data load during which time the applied load is high but
fairly constant (500-700 writes/sec).
> 
> I'm trying to understand the slowdown: there is no memory pressure, I've run every option
under nodetool to look for bottlenecks (tpstats, compactionStats, etc) and see none.  I'm
running with keycache and have about 98% hits.
> 
> What can I check next?
> Thanks!
> 
> Brian Tarbox


Mime
View raw message