incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamar Fraenkel <ta...@tok-media.com>
Subject Re: Lots of GCInspector.java on my cluster
Date Wed, 04 Jul 2012 19:07:46 GMT
Thanks.
I thought my problems may be related to the second leap and I ran sudo date
-s "`date -u`" on all nodes. Things have improved much in the last 24 hours.

*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

tamar@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956





On Wed, Jul 4, 2012 at 1:39 PM, aaron morton <aaron@thelastpickle.com>wrote:

> High CPU can be http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs
>
> memory usage looks ok http://wiki.apache.org/cassandra/FAQ#mmap
>
> Cheers
>
>
>   -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 3/07/2012, at 6:49 PM, Tamar Fraenkel wrote:
>
> Hi!
> I have a Cassandra cluster on Amazon EC2 Datastax AMIs with 3 nodes and
> replication factor of 2.
> As of July 1st the cluster is very slow and seems to be loaded.
>
> Running top I get:
>
> top - 06:40:58 up 99 days, 21:30,  2 users,  load average: 12.45, 13.37,
> 14.01
> Tasks: 102 total,   1 running, 101 sleeping,   0 stopped,   0 zombie
> Cpu(s): 21.0%us,  9.8%sy,  0.0%ni,  2.2%id,  0.0%wa,  0.4%hi,  0.4%si,
> 66.3%st
> Mem:   7647812k total,  7135752k used,   512060k free,    60668k buffers
> Swap:        0k total,        0k used,        0k free,  4234008k cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 18729 cassandr  20   0 8866m 2.5g 275m S  108 34.0   1415:37 jsvc
> 21798 root      20   0  478m 217m 9760 S   44  2.9   1743:47 java
>     3 root      20   0     0    0    0 S   20  0.0 564:59.09 ksoftirqd/0
>
>
>
> These are the cassandra processes
> *ps -ef | grep cassandra*
> root     18727     1  0 Jul02 ?        00:00:00 jsvc.exec -user cassandra-home /usr/lib/jvm/java-6-sun/jre/bin/../
-pidfile /var/run/
> cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp
> /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra
> /lib/avro-1.4.0-fixes.jar:/usr/share/cassandra
> /lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra
> /lib/commons-cli-1.1.jar:/usr/share/cassandra
> /lib/commons-codec-1.2.jar:/usr/share/cassandra
> /lib/commons-lang-2.4.jar:/usr/share/cassandra
> /lib/compress-lzf-0.8.4.jar:/usr/share/cassandra
> /lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra
> /lib/guava-r08.jar:/usr/share/cassandra
> /lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra
> /lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra
> /lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra
> /lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/
> cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra
> /lib/json-simple-1.1.jar:/usr/share/cassandra
> /lib/libthrift-0.6.jar:/usr/share/cassandra
> /lib/log4j-1.2.16.jar:/usr/share/cassandra
> /lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra
> /lib/slf4j-api-1.6.1.jar:/usr/share/cassandra
> /lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra
> /lib/snakeyaml-1.6.jar:/usr/share/cassandra
> /lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra
> /apache-cassandra-1.0.8.jar:/usr/share/cassandra
> /apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra
> /apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/
> java/commons-daemon.jar -Dlog4j.configuration=log4j-server.properties
> -XX:HeapDumpPath=/var/lib/cassandra/java_1341216340.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log -ea
> -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M
> -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
> -Djava.rmi.server.hostname=10.34.158.33
> -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra
> .thrift.CassandraDaemon
> 108      18729 18727 99 Jul02 ?        23:26:48 jsvc.exec -user cassandra-home /usr/lib/jvm/java-6-sun/jre/bin/../
-pidfile /var/run/
> cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp
> /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra
> /lib/avro-1.4.0-fixes.jar:/usr/share/cassandra
> /lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra
> /lib/commons-cli-1.1.jar:/usr/share/cassandra
> /lib/commons-codec-1.2.jar:/usr/share/cassandra
> /lib/commons-lang-2.4.jar:/usr/share/cassandra
> /lib/compress-lzf-0.8.4.jar:/usr/share/cassandra
> /lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra
> /lib/guava-r08.jar:/usr/share/cassandra
> /lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra
> /lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra
> /lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra
> /lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/
> cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra
> /lib/json-simple-1.1.jar:/usr/share/cassandra
> /lib/libthrift-0.6.jar:/usr/share/cassandra
> /lib/log4j-1.2.16.jar:/usr/share/cassandra
> /lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra
> /lib/slf4j-api-1.6.1.jar:/usr/share/cassandra
> /lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra
> /lib/snakeyaml-1.6.jar:/usr/share/cassandra
> /lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra
> /apache-cassandra-1.0.8.jar:/usr/share/cassandra
> /apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra
> /apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/
> java/commons-daemon.jar -Dlog4j.configuration=log4j-server.properties
> -XX:HeapDumpPath=/var/lib/cassandra/java_1341216340.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log -ea
> -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M
> -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
> -Djava.rmi.server.hostname=10.34.158.33
> -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra
> .thrift.CassandraDaemon
>
> Attached is the cfstats from node #1.
> Yesterday I restarted all the cluster, and it improved things a bit, but
> still memory consumption is high and things are slower than they used to be
> + many GC runs.
> OpsCenter is running on node#1 and probes on other two nodes.
>
> Any help would be greatly appreciated!
> Thanks,
>
> *Tamar Fraenkel *
> Senior Software Engineer, TOK Media
>
> <tokLogo.png>
>
>
> tamar@tok-media.com
> Tel:   +972 2 6409736
> Mob:  +972 54 8356490
> Fax:   +972 2 5612956
>
>
>
>  <cfstats4>
>
>
>

Mime
View raw message