cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: High CPU usage on all nodes without any read or write
Date Wed, 07 Jul 2010 16:17:19 GMT
> It runs correctly during several days. Last night, we started to have timeout exception
on insert and high cpu load on all nodes.
> We stopped inserts. But the CPU remains high (without any insert or read).

Has data been written to the cluster faster than background compaction
is proceeding? If so you may see cassandra eating CPU (and doing I/O)
in the background for extended periods of time even after you stop
sending requests to it.

If this is what is happening it should be visible in the log that it's
doing compaction, and you should see that the data directories contain
lots of files (unless it's just now catching up) rather than the
fairly few expectation when compaction is up to speed.

Also consider that even if you're not writing faster than it can
handle, if you have lots of data in total, the bigger compactions will
take a considerable mount of time so you may see CPU+disk activity for
long periods even if all is otherwise well.

Of course you say your're seeing timeouts. Is is possible these are
timeouts that happen during compaction in general? What kind of
latency are we talking about (a few extra hundre millis or several
seconds?) and is there a correlation between the timeouts and lots of
data being flushed to disk (iostat -x -k 1)?

/ Peter Schuller

View raw message