cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Anybody experience one Cassandra server locking up?
Date Wed, 19 Aug 2009 16:12:24 GMT
sounds like you are exhausting the memory on that instance and it is
going into "GC swap" trying to free enough to continue.  this is very
easy to do on 0.3 -- try upgrading to the 0.4 beta if you are using
0.3.

On Tue, Aug 18, 2009 at 3:36 PM, Brian Frank
Cooper<cooperb@yahoo-inc.com> wrote:
> Hi folks,
>
>
>
> I have been loading a 6-server Cassandra cluster with 1KB records. After a
> few million inserts, the insert rate drops dramatically. After
> investigation, one of the Cassandra servers seems to be in a bad state,
> using 100% of one core on an 8-core machine, and 0% on the other cores.
> Inserts to this box have completely stopped, and the inserts to the other
> boxes have slowed way down (more than a factor of 10 slower.) A “kill” or
> “kill -3” to the bad java process does nothing; I have to use “kill -9” to
> stop it. Has anybody experienced anything like this?
>
>
>
> Additional info:
>
>
>
> The servers are 8 core, 8GB servers. I am running 64 bit java 1.6, and here
> are the JVM options:
>
>
>
> # Arguments to pass to the JVM
>
> JVM_OPTS=" \
>
>         -ea \
>
>         -Xdebug \
>
>         -Xrunjdwp:transport=dt_socket,server=y,address=8888,suspend=n \
>
>         -Xms128M \
>
>         -Xmx6G \
>
>         -XX:SurvivorRatio=8 \
>
>         -XX:TargetSurvivorRatio=90 \
>
>         -XX:+AggressiveOpts \
>
>         -XX:+UseParNewGC \
>
>         -XX:+UseConcMarkSweepGC \
>
>         -XX:CMSInitiatingOccupancyFraction=1 \
>
>         -XX:+CMSParallelRemarkEnabled \
>
>         -XX:+HeapDumpOnOutOfMemoryError \
>
>         -Dcom.sun.management.jmxremote.port=8080 \
>
>         -Dcom.sun.management.jmxremote.ssl=false \
>
>         -Dcom.sun.management.jmxremote.authenticate=false"
>
>
>
> (standard options from the Cassandra distribution, except for the 6GB of
> heap space.)
>
>
>
> Replication factor is 1 (this is just a test, not a production setup) and
> memtable size is set to 1GB.
>
>
>
> Thanks…
>
>
>
> brian

Mime
View raw message