incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Hendry <dan.hendry.j...@gmail.com>
Subject Severe Reliability Problems - 0.7 RC2
Date Mon, 20 Dec 2010 18:48:58 GMT
I have been having severe and strange reliability problems within my
Cassandra cluster. This weekend, all four of my nodes were down at once.
Even now I am loosing one every few hours. I have attached output from all
the system monitoring commands I can think of.

What seems to happen is that the java process locks up and sits and has 100%
system cpu usage (but no user-CPU) (there are 8 cores so 100%=1/8 total
capacity). JMX freezes and the node effectively dies, but there is typically
nothing unusual in the Cassandra logs. About the only thing which seems to
be correlated is the flushing of memtables tables. One of the strangest
stats I am getting when in this state is memory paging: 3727168.00 pages
scanned/second (see sar -B output). Occasionally, if I leave the process
alone (~1 h) it recovers (maybe 1 in 5 times), otherwise the only way to
terminate the Cassandra process is with a kill -9. When this happens,
Cassandra memory usage (as reported by JMX before it dies) is
also reasonable (ex 6 GB out of 12 GB heap and 24 GB system).

This feels more like a system level problem than a Cassandra problem so I
have tried diversifying my cluster, one node runs Ubuntu 10.10, the other
three 10.04. One runs OpenJDK (1.6.0_20), the rest run Sun JDK (1.6.0_22).
Neither change seems be correlated with the problem. These are pretty much
stock ubuntu installs so nothing special on that front.

Now this has been a relatively sudden development and I can potentially
attribute it to a few things:
1. Upgrading to RC2
2. Ever increasing amounts of data (there is less than 100 gb per node so
this should not be the problem).
3. Migrating from a set of machines where data+commit log directories were
on four small raid 5 hds to machines with two 500 gig drives: one for data
and one for commitlog + os. I have seen more IO wait on these new machines.
But they have the same memory and system settings.

I am about at my wits end on this one, any help would be appreciated.

Mime
View raw message