cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@almaden.ibm.com>
Subject Re: Anybody experience one Cassandra server locking up?
Date Wed, 19 Aug 2009 15:53:55 GMT
Do you see any exceptions in the Cassandra log?

Jun
IBM Almaden Research Center
K55/B1, 650 Harry Road, San Jose, CA  95120-6099

junrao@almaden.ibm.com



|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Brian Frank Cooper <cooperb@yahoo-inc.com>                                      
                                                                 |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"cassandra-user@incubator.apache.org" <cassandra-user@incubator.apache.org>     
                                                                 |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |08/18/2009 03:37 PM                                                                   
                                                           |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Anybody experience one Cassandra server locking up?                                   
                                                           |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|





Hi folks,

I have been loading a 6-server Cassandra cluster with 1KB records. After a
few million inserts, the insert rate drops dramatically. After
investigation, one of the Cassandra servers seems to be in a bad state,
using 100% of one core on an 8-core machine, and 0% on the other cores.
Inserts to this box have completely stopped, and the inserts to the other
boxes have slowed way down (more than a factor of 10 slower.) A “kill” or
“kill -3” to the bad java process does nothing; I have to use “kill -9” to
stop it. Has anybody experienced anything like this?

Additional info:

The servers are 8 core, 8GB servers. I am running 64 bit java 1.6, and here
are the JVM options:

# Arguments to pass to the JVM
JVM_OPTS=" \
        -ea \
        -Xdebug \
        -Xrunjdwp:transport=dt_socket,server=y,address=8888,suspend=n \
        -Xms128M \
        -Xmx6G \
        -XX:SurvivorRatio=8 \
        -XX:TargetSurvivorRatio=90 \
        -XX:+AggressiveOpts \
        -XX:+UseParNewGC \
        -XX:+UseConcMarkSweepGC \
        -XX:CMSInitiatingOccupancyFraction=1 \
        -XX:+CMSParallelRemarkEnabled \
        -XX:+HeapDumpOnOutOfMemoryError \
        -Dcom.sun.management.jmxremote.port=8080 \
        -Dcom.sun.management.jmxremote.ssl=false \
        -Dcom.sun.management.jmxremote.authenticate=false"

(standard options from the Cassandra distribution, except for the 6GB of
heap space.)

Replication factor is 1 (this is just a test, not a production setup) and
memtable size is set to 1GB.

Thanks…

brian
Mime
View raw message