incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Tata <sandeep.t...@gmail.com>
Subject Re: Anybody experience one Cassandra server locking up?
Date Wed, 19 Aug 2009 20:29:05 GMT
Brian,

Are you guys planning to run workloads at Yahoo to compare Cassandra and PNUTS?
We'd be curious to see what you learn with the 0.4/trunk code.

Sandeep

On Wed, Aug 19, 2009 at 10:20 AM, Brian Frank
Cooper<cooperb@yahoo-inc.com> wrote:
> Probably you are right; after Jun's response I looked in the log and saw an out of memory
exception. I'll try the 0.4 beta...
>
> Thanks!
>
> brian
>
> -----Original Message-----
> From: Jonathan Ellis [mailto:jbellis@gmail.com]
> Sent: Wednesday, August 19, 2009 9:12 AM
> To: cassandra-user@incubator.apache.org
> Subject: Re: Anybody experience one Cassandra server locking up?
>
> sounds like you are exhausting the memory on that instance and it is
> going into "GC swap" trying to free enough to continue.  this is very
> easy to do on 0.3 -- try upgrading to the 0.4 beta if you are using
> 0.3.
>
> On Tue, Aug 18, 2009 at 3:36 PM, Brian Frank
> Cooper<cooperb@yahoo-inc.com> wrote:
>> Hi folks,
>>
>>
>>
>> I have been loading a 6-server Cassandra cluster with 1KB records. After a
>> few million inserts, the insert rate drops dramatically. After
>> investigation, one of the Cassandra servers seems to be in a bad state,
>> using 100% of one core on an 8-core machine, and 0% on the other cores.
>> Inserts to this box have completely stopped, and the inserts to the other
>> boxes have slowed way down (more than a factor of 10 slower.) A "kill" or
>> "kill -3" to the bad java process does nothing; I have to use "kill -9" to
>> stop it. Has anybody experienced anything like this?
>>
>>
>>
>> Additional info:
>>
>>
>>
>> The servers are 8 core, 8GB servers. I am running 64 bit java 1.6, and here
>> are the JVM options:
>>
>>
>>
>> # Arguments to pass to the JVM
>>
>> JVM_OPTS=" \
>>
>>         -ea \
>>
>>         -Xdebug \
>>
>>         -Xrunjdwp:transport=dt_socket,server=y,address=8888,suspend=n \
>>
>>         -Xms128M \
>>
>>         -Xmx6G \
>>
>>         -XX:SurvivorRatio=8 \
>>
>>         -XX:TargetSurvivorRatio=90 \
>>
>>         -XX:+AggressiveOpts \
>>
>>         -XX:+UseParNewGC \
>>
>>         -XX:+UseConcMarkSweepGC \
>>
>>         -XX:CMSInitiatingOccupancyFraction=1 \
>>
>>         -XX:+CMSParallelRemarkEnabled \
>>
>>         -XX:+HeapDumpOnOutOfMemoryError \
>>
>>         -Dcom.sun.management.jmxremote.port=8080 \
>>
>>         -Dcom.sun.management.jmxremote.ssl=false \
>>
>>         -Dcom.sun.management.jmxremote.authenticate=false"
>>
>>
>>
>> (standard options from the Cassandra distribution, except for the 6GB of
>> heap space.)
>>
>>
>>
>> Replication factor is 1 (this is just a test, not a production setup) and
>> memtable size is set to 1GB.
>>
>>
>>
>> Thanks.
>>
>>
>>
>> brian
>

Mime
View raw message