incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Tarbox <tar...@cabotresearch.com>
Subject read performance plumetted
Date Fri, 12 Oct 2012 13:33:00 GMT
I have a two node cluster hosting a 45 gig dataset.  I periodically have to
read a high fraction (20% or so) of my 'rows', grabbing a few thousand at a
time and then processing them.

This used to result in about 300-500 reads a second which seemed quite
good.  Recently that number has plummeted to 20-50 reads a second.  The
obvious question is what did I change?  I certainly added more
data....bringing my total load from 38 or so gig to 45 or so gig but its
hard to imagine that causing this problem.  The shape of my data has not
changed and I haven't changed any cassandra configuration.

Running nodetool tpstats I'm for the first time ever seeing entries under
ReadStage Active and Pending  which correlates with slow reads.
Running iostat I'm seeing a significant (10-50%) of iowait where I
previously never saw higher than 1-2%

I ran a full compaction on the relevant CF (which took 3.5 hours) to no
avail.

Any suggestions on where I can look next?

Thanks.

Mime
View raw message