Do you see any exceptions in the Cassandra log?
IBM Almaden Research Center
K55/B1, 650 Harry Road, San Jose, CA 95120-6099
Brian Frank Cooper ---08/18/2009 03:37:40 PM---Hi folks, I have been loading a 6-server Cassandra cluster with 1KB records. After a few million ins
Brian Frank Cooper <firstname.lastname@example.org>
08/18/2009 03:37 PM
Anybody experience one Cassandra server locking up?
I have been loading a 6-server Cassandra cluster with 1KB records. After a few million inserts, the insert rate drops dramatically. After investigation, one of the Cassandra servers seems to be in a bad state, using 100% of one core on an 8-core machine, and 0% on the other cores. Inserts to this box have completely stopped, and the inserts to the other boxes have slowed way down (more than a factor of 10 slower.) A “kill” or “kill -3” to the bad java process does nothing; I have to use “kill -9” to stop it. Has anybody experienced anything like this?
The servers are 8 core, 8GB servers. I am running 64 bit java 1.6, and here are the JVM options:
# Arguments to pass to the JVM
(standard options from the Cassandra distribution, except for the 6GB of heap space.)
Replication factor is 1 (this is just a test, not a production setup) and memtable size is set to 1GB.