I am running a cassandra cluster of 6 nodes running RHEL6 virtualized by
ESXi 5.0. Each VM is configured with 20GB of ram and 12 cores. Our test
setup performs about 3000 inserts per second. The cassandra data partition
is on a XFS filesystem mounted with options
(noatime,nodiratime,nobarrier,logbufs=8). We have no swap enabled on the VMs
and the vm.swappiness set to 0. To avoid any contention issues our cassandra
VMs are not running any other application other than cassandra.
The test runs fine for about 12 hours or so. After that the performance
starts to degrade to about 1500 inserts per sec. By 18-20 hours the inserts
go down to 300 per sec.
if i do a truncate, it starts clean, runs for a few hours (not as clean as
rebooting).
We find a direct correlation between kswapd kicking in after 12 hours or so
and the performance degradation. If i look at the cached memory it is
close to 10G. I am not getting a OOM error in cassandra. So looks like we
are not running out of memory. Can some one explain if we can optimize this
so that kswapd doesn't kick in.
Our top output shows
top - 16:23:54 up 2 days, 23:17, 4 users, load average: 2.21, 2.08, 2.02
Tasks: 213 total, 1 running, 212 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.6%us, 0.8%sy, 0.0%ni, 90.9%id, 6.3%wa, 0.0%hi, 0.2%si,
0.0%st
Mem: 20602812k total, 20320424k used, 282388k free, 1020k buffers
Swap: 0k total, 0k used, 0k free, 10145516k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2586 root 20 0 36.3g 17g 8.4g S 32.1 88.9 8496:37 java
java output
root 2453 1 99 Sep30 pts/0 9-13:51:38 java -ea
-javaagent:./apache-cassandra-0.8.6/bin/../lib/jamm-0.2.2.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms10059M -Xmx10059M
-Xmn1200M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=10.19.104.14 -Djava.net.preferIPv4Stack=true
-Dlog4j.configuration=log4j-server.properties
-Dlog4j.defaultInitOverride=true -cp
./apache-cassandra-0.8.6/bin/../conf:./apache-cassandra-0.8.6/bin/../build/classes/main:./apache-cassandra-0.8.6/bin/../build/classes/thrift:./apache-cassandra-0.8.6/bin/../lib/antlr-3.2.jar:./apache-cassandra-0.8.6/bin/../lib/apache-cassandra-0.8.6.jar:./apache-cassandra-0.8.6/bin/../lib/apache-cassandra-thrift-0.8.6.jar:./apache-cassandra-0.8.6/bin/../lib/avro-1.4.0-fixes.jar:./apache-cassandra-0.8.6/bin/../lib/avro-1.4.0-sources-fixes.jar:./apache-cassandra-0.8.6/bin/../lib/commons-cli-1.1.jar:./apache-cassandra-0.8.6/bin/../lib/commons-codec-1.2.jar:./apache-cassandra-0.8.6/bin/../lib/commons-collections-3.2.1.jar:./apache-cassandra-0.8.6/bin/../lib/commons-lang-2.4.jar:./apache-cassandra-0.8.6/bin/../lib/concurrentlinkedhashmap-lru-1.1.jar:./apache-cassandra-0.8.6/bin/../lib/guava-r08.jar:./apache-cassandra-0.8.6/bin/../lib/high-scale-lib-1.1.2.jar:./apache-cassandra-0.8.6/bin/../lib/jackson-core-asl-1.4.0.jar:./apache-cassandra-0.8.6/bin/../lib/jackson-mapper-asl-1.4.0.jar:./apache-cassandra-0.8.6/bin/../lib/jamm-0.2.2.jar:./apache-cassandra-0.8.6/bin/../lib/jline-0.9.94.jar:./apache-cassandra-0.8.6/bin/../lib/json-simple-1.1.jar:./apache-cassandra-0.8.6/bin/../lib/libthrift-0.6.jar:./apache-cassandra-0.8.6/bin/../lib/log4j-1.2.16.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-examples.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-impl.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-jmx.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-remote.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-rimpl.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-rjmx.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-tools.jar:./apache-cassandra-0.8.6/bin/../lib/servlet-api-2.5-20081211.jar:./apache-cassandra-0.8.6/bin/../lib/slf4j-api-1.6.1.jar:./apache-cassandra-0.8.6/bin/../lib/slf4j-log4j12-1.6.1.jar:./apache-cassandra-0.8.6/bin/../lib/snakeyaml-1.6.jar
org.apache.cassandra.thrift.CassandraDaemon
Ring output
[root@CAP4-CNode4 apache-cassandra-0.8.6]# ./bin/nodetool -h 127.0.0.1 ring
Address DC Rack Status State Load Owns
Token
141784319550391026443072753096570088105
10.19.104.11 datacenter1 rack1 Up Normal 19.92 GB
16.67% 0
10.19.104.12 datacenter1 rack1 Up Normal 19.3 GB
16.67% 28356863910078205288614550619314017621
10.19.104.13 datacenter1 rack1 Up Normal 18.57 GB
16.67% 56713727820156410577229101238628035242
10.19.104.14 datacenter1 rack1 Up Normal 19.34 GB
16.67% 85070591730234615865843651857942052863
10.19.105.11 datacenter1 rack1 Up Normal 19.88 GB
16.67% 113427455640312821154458202477256070484
10.19.105.12 datacenter1 rack1 Up Normal 20 GB
16.67% 141784319550391026443072753096570088105
[root@CAP4-CNode4 apache-cassandra-0.8.6]#
|