incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "B. Todd Burruss" <>
Subject Re: read performance plumetted
Date Fri, 12 Oct 2012 17:25:51 GMT
did the amount of data finally exceed your per machine RAM capacity?
is it the same 20% each time you read?  or do your periodic reads
eventually work through the entire dataset?

if you are essentially table scanning your data set, and the size
exceeds available RAM, then a degradation like that isn't crazy.  and
this is indicated with your iowait %%

On Fri, Oct 12, 2012 at 6:33 AM, Brian Tarbox <> wrote:
> I have a two node cluster hosting a 45 gig dataset.  I periodically have to
> read a high fraction (20% or so) of my 'rows', grabbing a few thousand at a
> time and then processing them.
> This used to result in about 300-500 reads a second which seemed quite good.
> Recently that number has plummeted to 20-50 reads a second.  The obvious
> question is what did I change?  I certainly added more data....bringing my
> total load from 38 or so gig to 45 or so gig but its hard to imagine that
> causing this problem.  The shape of my data has not changed and I haven't
> changed any cassandra configuration.
> Running nodetool tpstats I'm for the first time ever seeing entries under
> ReadStage Active and Pending  which correlates with slow reads.
> Running iostat I'm seeing a significant (10-50%) of iowait where I
> previously never saw higher than 1-2%
> I ran a full compaction on the relevant CF (which took 3.5 hours) to no
> avail.
> Any suggestions on where I can look next?
> Thanks.

View raw message