incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramesh Natarajan <rames...@gmail.com>
Subject cassandra performance degrades after 12 hours
Date Mon, 03 Oct 2011 17:12:46 GMT
I am running a cassandra cluster of  6 nodes running RHEL6 virtualized by
ESXi 5.0.  Each VM is configured with 20GB of ram and 12 cores. Our test
setup performs about 3000  inserts per second.  The cassandra data partition
is on a XFS filesystem mounted with options
(noatime,nodiratime,nobarrier,logbufs=8). We have no swap enabled on the VMs
and the vm.swappiness set to 0. To avoid any contention issues our cassandra
VMs are not running any other application other than cassandra.

The test runs fine for about 12 hours or so. After that the performance
starts to degrade to about 1500 inserts per sec. By 18-20 hours the inserts
go down to 300 per sec.
if i do a truncate, it starts clean, runs for a few hours (not as clean as
rebooting).

We find a direct correlation between kswapd kicking in after 12 hours or so
and the performance degradation.   If i look at the cached memory it is
close to 10G.  I am not getting a OOM error in cassandra. So looks like we
are not running out of memory. Can some one explain if we can optimize this
so that kswapd doesn't kick in.


Our top output shows
top - 16:23:54 up 2 days, 23:17,  4 users,  load average: 2.21, 2.08, 2.02
Tasks: 213 total,   1 running, 212 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.6%us,  0.8%sy,  0.0%ni, 90.9%id,  6.3%wa,  0.0%hi,  0.2%si,
 0.0%st
Mem:  20602812k total, 20320424k used,   282388k free,     1020k buffers
Swap:        0k total,        0k used,        0k free, 10145516k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

 2586 root      20   0 36.3g  17g 8.4g S 32.1 88.9   8496:37 java


java output

root      2453     1 99 Sep30 pts/0    9-13:51:38 java -ea
-javaagent:./apache-cassandra-0.8.6/bin/../lib/jamm-0.2.2.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms10059M -Xmx10059M
-Xmn1200M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=10.19.104.14 -Djava.net.preferIPv4Stack=true
-Dlog4j.configuration=log4j-server.properties
-Dlog4j.defaultInitOverride=true -cp
./apache-cassandra-0.8.6/bin/../conf:./apache-cassandra-0.8.6/bin/../build/classes/main:./apache-cassandra-0.8.6/bin/../build/classes/thrift:./apache-cassandra-0.8.6/bin/../lib/antlr-3.2.jar:./apache-cassandra-0.8.6/bin/../lib/apache-cassandra-0.8.6.jar:./apache-cassandra-0.8.6/bin/../lib/apache-cassandra-thrift-0.8.6.jar:./apache-cassandra-0.8.6/bin/../lib/avro-1.4.0-fixes.jar:./apache-cassandra-0.8.6/bin/../lib/avro-1.4.0-sources-fixes.jar:./apache-cassandra-0.8.6/bin/../lib/commons-cli-1.1.jar:./apache-cassandra-0.8.6/bin/../lib/commons-codec-1.2.jar:./apache-cassandra-0.8.6/bin/../lib/commons-collections-3.2.1.jar:./apache-cassandra-0.8.6/bin/../lib/commons-lang-2.4.jar:./apache-cassandra-0.8.6/bin/../lib/concurrentlinkedhashmap-lru-1.1.jar:./apache-cassandra-0.8.6/bin/../lib/guava-r08.jar:./apache-cassandra-0.8.6/bin/../lib/high-scale-lib-1.1.2.jar:./apache-cassandra-0.8.6/bin/../lib/jackson-core-asl-1.4.0.jar:./apache-cassandra-0.8.6/bin/../lib/jackson-mapper-asl-1.4.0.jar:./apache-cassandra-0.8.6/bin/../lib/jamm-0.2.2.jar:./apache-cassandra-0.8.6/bin/../lib/jline-0.9.94.jar:./apache-cassandra-0.8.6/bin/../lib/json-simple-1.1.jar:./apache-cassandra-0.8.6/bin/../lib/libthrift-0.6.jar:./apache-cassandra-0.8.6/bin/../lib/log4j-1.2.16.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-examples.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-impl.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-jmx.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-remote.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-rimpl.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-rjmx.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-tools.jar:./apache-cassandra-0.8.6/bin/../lib/servlet-api-2.5-20081211.jar:./apache-cassandra-0.8.6/bin/../lib/slf4j-api-1.6.1.jar:./apache-cassandra-0.8.6/bin/../lib/slf4j-log4j12-1.6.1.jar:./apache-cassandra-0.8.6/bin/../lib/snakeyaml-1.6.jar
org.apache.cassandra.thrift.CassandraDaemon



Ring output

[root@CAP4-CNode4 apache-cassandra-0.8.6]# ./bin/nodetool -h 127.0.0.1 ring
Address         DC          Rack        Status State   Load            Owns
   Token

   141784319550391026443072753096570088105
10.19.104.11    datacenter1 rack1       Up     Normal  19.92 GB
 16.67%  0
10.19.104.12    datacenter1 rack1       Up     Normal  19.3 GB
16.67%  28356863910078205288614550619314017621
10.19.104.13    datacenter1 rack1       Up     Normal  18.57 GB
 16.67%  56713727820156410577229101238628035242
10.19.104.14    datacenter1 rack1       Up     Normal  19.34 GB
 16.67%  85070591730234615865843651857942052863
10.19.105.11    datacenter1 rack1       Up     Normal  19.88 GB
 16.67%  113427455640312821154458202477256070484
10.19.105.12    datacenter1 rack1       Up     Normal  20 GB
16.67%  141784319550391026443072753096570088105
[root@CAP4-CNode4 apache-cassandra-0.8.6]#

Mime
View raw message