cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: keycache persisted to disk ?
Date Mon, 13 Feb 2012 08:21:17 GMT
> I actually has the opposite 'problem'. I have a pair of servers that have
> been static since mid last week, but have seen performance vary
> significantly (x10) for exactly the same query. I hypothesised it was
> various caches so I shut down Cassandra, flushed the O/S buffer cache and
> then bought it back up. The performance wasn't significantly different to
> the pre-flush performance

I don't get this thread at all :)

Why would restarting with clean caches be expected to *improve*
performance? And why is key cache loading involved other than to delay
start-up and hopefully pre-populating caches for better (not worse)
performance?

If you want to figure out why queries seem to be slow relative to
normal, you'll need to monitor the behavior of the nodes. Look at disk
I/O statistics primarily (everyone reading this running Cassandra who
aren't intimately familiar with "iostat -x -k 1" should go and read up
on it right away; make sure you understand the utilization and avg
queue size columns), CPU usage, weather compaction is happening, etc.

One easy way to see sudden bursts of poor behavior is to be heavily
reliant on cache, and then have sudden decreases in performance due to
compaction evicting data from page cache while also generating more
I/O.

But that's total speculation. It is also the case that you cannot
expect consistent performance on EC2 and that might be it.

But my #1 advise: Log into the node while it is being slow, and
observe. Figure out what the bottleneck is. iostat, top, nodetool
tpstats, nodetool netstats, nodetool compactionstats.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)

Mime
View raw message