incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Lots of GCInspector.java on my cluster
Date Wed, 04 Jul 2012 10:39:42 GMT
High CPU can be http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs

memory usage looks ok http://wiki.apache.org/cassandra/FAQ#mmap

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 3/07/2012, at 6:49 PM, Tamar Fraenkel wrote:

> Hi!
> I have a Cassandra cluster on Amazon EC2 Datastax AMIs with 3 nodes and replication factor
of 2.
> As of July 1st the cluster is very slow and seems to be loaded.
> 
> Running top I get:
> 
> top - 06:40:58 up 99 days, 21:30,  2 users,  load average: 12.45, 13.37, 14.01
> Tasks: 102 total,   1 running, 101 sleeping,   0 stopped,   0 zombie
> Cpu(s): 21.0%us,  9.8%sy,  0.0%ni,  2.2%id,  0.0%wa,  0.4%hi,  0.4%si, 66.3%st
> Mem:   7647812k total,  7135752k used,   512060k free,    60668k buffers
> Swap:        0k total,        0k used,        0k free,  4234008k cached
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 18729 cassandr  20   0 8866m 2.5g 275m S  108 34.0   1415:37 jsvc
> 21798 root      20   0  478m 217m 9760 S   44  2.9   1743:47 java
>     3 root      20   0     0    0    0 S   20  0.0 564:59.09 ksoftirqd/0
> 
> 
> 
> These are the cassandra processes
> ps -ef | grep cassandra
> root     18727     1  0 Jul02 ?        00:00:00 jsvc.exec -user cassandra -home /usr/lib/jvm/java-6-sun/jre/bin/../
-pidfile /var/run/cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.8.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar
-Dlog4j.configuration=log4j-server.properties -XX:HeapDumpPath=/var/lib/cassandra/java_1341216340.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M -Xmn200M -XX:+HeapDumpOnOutOfMemoryError
-Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=10.34.158.33 -Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
org.apache.cassandra.thrift.CassandraDaemon
> 108      18729 18727 99 Jul02 ?        23:26:48 jsvc.exec -user cassandra -home /usr/lib/jvm/java-6-sun/jre/bin/../
-pidfile /var/run/cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.8.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar
-Dlog4j.configuration=log4j-server.properties -XX:HeapDumpPath=/var/lib/cassandra/java_1341216340.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M -Xmn200M -XX:+HeapDumpOnOutOfMemoryError
-Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=10.34.158.33 -Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
org.apache.cassandra.thrift.CassandraDaemon
> 
> Attached is the cfstats from node #1.
> Yesterday I restarted all the cluster, and it improved things a bit, but still memory
consumption is high and things are slower than they used to be + many GC runs.
> OpsCenter is running on node#1 and probes on other two nodes.
> 
> Any help would be greatly appreciated!
> Thanks,
> 
> Tamar Fraenkel 
> Senior Software Engineer, TOK Media 
> 
> <tokLogo.png>
> 
> tamar@tok-media.com
> Tel:   +972 2 6409736 
> Mob:  +972 54 8356490 
> Fax:   +972 2 5612956 
> 
> 
> 
> <cfstats4>


Mime
View raw message