incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Lee <James....@metaswitch.com>
Subject RE: Cassandra periodically stops responding to write requests under load
Date Fri, 14 Jun 2013 17:10:42 GMT
Hi Rob,

Thanks for the reply.  To answer your questions below:

I'm using the following JVM:
java version "1.7.0_10"
Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)

Stock Cassandra version 1.2.2

Heap settings are as follows (the system has 48GB of physical RAM, with over 2/3 of it unused
while Cassandra is running): -Xms8192M -Xmx8192M -Xmn2048M

I don't see any GC logs around the time of the slowness.

For now I'm testing with a single node only.  I'm expecting to have multiple nodes with all
data fully replicated (eg. 2 nodes, replication factor 2) so am testing with a single node
to check performance in the case where all other nodes have failed.  I'd like to get away
with as few nodes as possible ideally...

I've put the full commandline of the running java process below, in case any of it is relevant:
/usr/jdk1.7.0_10/bin/java -ea -javaagent:/dsc-cassandra-1.2.2/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M -Xmn2048M -XX:+HeapDumpOnOutOfMemoryError
-Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp /dsc-cassandra-1.2.2/bin/../conf:/dsc-cassandra-1.2.2/bin/../build/classes/main:/dsc-cassandra-1.2.2/bin/../build/classes/thrift:/dsc-cassandra-1.2.2/bin/../lib/antlr-3.2.jar:/dsc-cassandra-1.2.2/bin/../lib/apache-cassandra-1.2.2.jar:/dsc-cassandra-1.2.2/bin/../lib/apache-cassandra-clientutil-1.2.2.jar:/dsc-cassandra-1.2.2/bin/../lib/apache-cassandra-thrift-1.2.2.jar:/dsc-cassandra-1.2.2/bin/../lib/avro-1.4.0-fixes.jar:/dsc-cassandra-1.2.2/bin/../lib/avro-1.4.0-sources-fixes.jar:/dsc-cassandra-1.2.2/bin/../lib/commons-cli-1.1.jar:/dsc-cassandra-1.2.2/bin/../lib/commons-codec-1.2.jar:/dsc-cassandra-1.2.2/bin/../lib/commons-lang-2.6.jar:/dsc-cassandra-1.2.2/bin/../lib/compress-lzf-0.8.4.jar:/dsc-cassandra-1.2.2/bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:/dsc-cassandra-1.2.2/bin/../lib/guava-13.0.1.jar:/dsc-cassandra-1.2.2/bin/../lib/high-scale-lib-1.1.2.jar:/dsc-cassandra-1.2.2/bin/../lib/jackson-core-asl-1.9.2.jar:/dsc-cassandra-1.2.2/bin/../lib/jackson-mapper-asl-1.9.2.jar:/dsc-cassandra-1.2.2/bin/../lib/jamm-0.2.5.jar:/dsc-cassandra-1.2.2/bin/../lib/jbcrypt-0.3m.jar:/dsc-cassandra-1.2.2/bin/../lib/jline-1.0.jar:/dsc-cassandra-1.2.2/bin/../lib/jna-3.4.0.jar:/dsc-cassandra-1.2.2/bin/../lib/json-simple-1.1.jar:/dsc-cassandra-1.2.2/bin/../lib/libthrift-0.7.0.jar:/dsc-cassandra-1.2.2/bin/../lib/log4j-1.2.16.jar:/dsc-cassandra-1.2.2/bin/../lib/lz4-1.1.0.jar:/dsc-cassandra-1.2.2/bin/../lib/metrics-core-2.0.3.jar:/dsc-cassandra-1.2.2/bin/../lib/netty-3.5.9.Final.jar:/dsc-cassandra-1.2.2/bin/../lib/servlet-api-2.5-20081211.jar:/dsc-cassandra-1.2.2/bin/../lib/slf4j-api-1.7.2.jar:/dsc-cassandra-1.2.2/bin/../lib/slf4j-log4j12-1.7.2.jar:/dsc-cassandra-1.2.2/bin/../lib/snakeyaml-1.6.jar:/dsc-cassandra-1.2.2/bin/../lib/snappy-java-1.0.4.1.jar:/dsc-cassandra-1.2.2/bin/../lib/snaptree-0.1.jar
org.apache.cassandra.service.CassandraDaemon

Thanks,
James

-----Original Message-----
From: Robert Coli [mailto:rcoli@eventbrite.com] 
Sent: 14 June 2013 17:17
To: user@cassandra.apache.org
Subject: Re: Cassandra periodically stops responding to write requests under load

On Fri, Jun 14, 2013 at 7:19 AM, James Lee <James.Lee@metaswitch.com> wrote:
>  I'm seeing generally good performance, but with periods where the 
> Cassandra node entirely stops responding to write requests for several 
> seconds at a time.  I don't have much experience of Cassandra 
> performance tuning, and would very much appreciate some pointers on what I can do to
improve matters.

It is relatively common for a Cassandra node to become unresponsive for a few seconds when
doing various things. However as one usually has multiple replicas for any given key, this
transient unavailability does not meaningfully impact overall availability. Pausing for more-than-a-few
seconds is relatively uncommon and probably does indicate either sub optimal configuration
or excessive workload.

> -- I've used a RAID array for the data directory to improve write 
> performance.  This significantly reduces the length of the slow period 
> (from ~10s to ~2s), but doesn't eliminate it.  I've tried RAID10 and 
> RAID0 using varying number of drives, but there doesn't seem to be a 
> significant difference between the two.

Do you see disk saturation when you're flushing? This statement suggests that you might be..

> -- I've used multiple drives for the data directory, symlinking the 
> directories for different keyspaces to different drives.  That didn't 
> improve things significantly compared to using a single drive.

I would not expect this to improve things if you are bounded on how quickly you can flush
from a single thread.

Stock questions :

1) What JVM?
2) What heap settings?
3) Do you also see GC logs around flush time?
4) Are you testing a single node only?

=Rob

Mime
View raw message