incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamar Fraenkel <ta...@tok-media.com>
Subject Lots of GCInspector.java on my cluster
Date Tue, 03 Jul 2012 06:49:10 GMT
Hi!
I have a Cassandra cluster on Amazon EC2 Datastax AMIs with 3 nodes and
replication factor of 2.
As of July 1st the cluster is very slow and seems to be loaded.

Running top I get:

top - 06:40:58 up 99 days, 21:30,  2 users,  load average: 12.45, 13.37,
14.01
Tasks: 102 total,   1 running, 101 sleeping,   0 stopped,   0 zombie
Cpu(s): 21.0%us,  9.8%sy,  0.0%ni,  2.2%id,  0.0%wa,  0.4%hi,  0.4%si,
66.3%st
Mem:   7647812k total,  7135752k used,   512060k free,    60668k buffers
Swap:        0k total,        0k used,        0k free,  4234008k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18729 cassandr  20   0 8866m 2.5g 275m S  108 34.0   1415:37 jsvc
21798 root      20   0  478m 217m 9760 S   44  2.9   1743:47 java
    3 root      20   0     0    0    0 S   20  0.0 564:59.09 ksoftirqd/0



These are the cassandra processes
*ps -ef | grep cassandra*
root     18727     1  0 Jul02 ?        00:00:00 jsvc.exec -user
cassandra-home /usr/lib/jvm/java-6-sun/jre/bin/../ -pidfile /var/run/
cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra
/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra
/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra
/lib/commons-cli-1.1.jar:/usr/share/cassandra
/lib/commons-codec-1.2.jar:/usr/share/cassandra
/lib/commons-lang-2.4.jar:/usr/share/cassandra
/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra
/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra
/lib/guava-r08.jar:/usr/share/cassandra
/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra
/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra
/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra
/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/
cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra
/lib/json-simple-1.1.jar:/usr/share/cassandra
/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/
cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra
/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra
/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra
/lib/snakeyaml-1.6.jar:/usr/share/cassandra
/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra
/apache-cassandra-1.0.8.jar:/usr/share/cassandra
/apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra
/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar
-Dlog4j.configuration=log4j-server.properties -XX:HeapDumpPath=/var/lib/
cassandra/java_1341216340.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log
-ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M
-Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
-Djava.rmi.server.hostname=10.34.158.33
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra
.thrift.CassandraDaemon
108      18729 18727 99 Jul02 ?        23:26:48 jsvc.exec -user
cassandra-home /usr/lib/jvm/java-6-sun/jre/bin/../ -pidfile /var/run/
cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra
/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra
/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra
/lib/commons-cli-1.1.jar:/usr/share/cassandra
/lib/commons-codec-1.2.jar:/usr/share/cassandra
/lib/commons-lang-2.4.jar:/usr/share/cassandra
/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra
/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra
/lib/guava-r08.jar:/usr/share/cassandra
/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra
/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra
/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra
/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/
cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra
/lib/json-simple-1.1.jar:/usr/share/cassandra
/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/
cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra
/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra
/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra
/lib/snakeyaml-1.6.jar:/usr/share/cassandra
/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra
/apache-cassandra-1.0.8.jar:/usr/share/cassandra
/apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra
/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar
-Dlog4j.configuration=log4j-server.properties -XX:HeapDumpPath=/var/lib/
cassandra/java_1341216340.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log
-ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M
-Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
-Djava.rmi.server.hostname=10.34.158.33
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra
.thrift.CassandraDaemon

Attached is the cfstats from node #1.
Yesterday I restarted all the cluster, and it improved things a bit, but
still memory consumption is high and things are slower than they used to be
+ many GC runs.
OpsCenter is running on node#1 and probes on other two nodes.

Any help would be greatly appreciated!
Thanks,

*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

tamar@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956

Mime
View raw message