cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: UnavailableException when data grows
Date Sat, 02 Oct 2010 20:45:29 GMT
> And I'm still getting UnavailableException and TimedOutException when there
> Cassandra daemon is doing either Compaction or Garbage collection...

Have you specifically correlated this? If so, which one, or both?

GC should not cause unavailable exceptions on a healthy cluster with
healthy nodes. If GC:s are causing nodes to not respond for so long
that they cause exceptions you may be swapping. Have you monitored to
see whether you are actively swapping in/out during GC:s (or at all
for that matter)? Excessive GC pause times should be logged by
Cassandra i the system log, regardless of JVM options. So, check your
cassandra log from messages from GCInspector about pause times.

I presume you are running with default JVM paramters?

WIth respect to compaction, I'm still interested, as in my original
response in this thread, what your data looks like and whether
compaction is CPU bound or I/O bound. It's quite possible that
compaction is having an adverse effect; if it is, I would suspect it
is due to disk I/O rather than CPU load (unless you are saturating
your cluster so that CPU load is the dominating factor). The primary
adverse effects expected by compaction, other than additional CPU
load, are:

* Generation of additional I/O which will directly affect normal traffic.
* Effects on the operating system buffer cache may increase the I/O
load resulting from normal traffic (thus making the preceding issue
even more significant).

Are the nodes that are slow to respond during compaction I/O bound?
Check with for example "iostat -x 1" (utilization value and average
queue size  being the most important columns).

There is some work planned/happening towards lessening the impact of
compactions, and it would be of interest to know the circumstances of
compaction problems that people do have.

-- 
/ Peter Schuller

Mime
View raw message