cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcin Cabaj <marcin.ca...@datasift.com>
Subject Re: memory usage spikes
Date Wed, 26 Mar 2014 17:52:41 GMT
You can try to dump memory mapping of the cassandra process during spike
using pmap, eg:
$ pmap -x <cassandra_pid>
and paste here.


On Wed, Mar 26, 2014 at 5:47 PM, prem yadav <ipremyadav@gmail.com> wrote:

> Thanks Don,
> Yes have followed those steps. Except jna. The version I am using is
> 3.2.4. The link you have shared is for Cassandra 2.0. I am using 1.1. Let
> me install jna 3.2.7 and see if that helps.
>
> Thanks
>
>
> On Wed, Mar 26, 2014 at 5:38 PM, Donald Smith <
> Donald.Smith@audiencescience.com> wrote:
>
>>  Prem,
>>
>>
>>
>> Did you follow the instructions at
>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html?scroll=reference_ds_sxl_gf3_2k
>>
>>
>>
>> And did you install jna-3.2.7.jar into /usr/share/java, as per
>> http://www.datastax.com/documentation/cassandra/2.0/mobile/cassandra/install/installJnaRHEL.html?
>>
>>
>>
>> Don
>>
>>
>>
>> *From:* prem yadav [mailto:ipremyadav@gmail.com]
>> *Sent:* Wednesday, March 26, 2014 10:36 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: memory usage spikes
>>
>>
>>
>> here:
>>
>>
>>
>> ps -p `/usr/java/jdk1.6.0_37/bin/jps | awk '/Dse/ {print $1}'` uww
>>
>>
>>
>> SER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>>
>> 497      20450  0.9 31.0 4727620 2502644 ?     SLl  06:55   3:28
>> /usr/java/jdk1.6.0_37//bin/java -ea
>> -javaagent:/usr/share/dse/cassandra/lib/jamm-0.2.5.jar
>> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1968M -Xmx1968M
>> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss190k -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
>> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
>> -Dcom.sun.management.jmxremote.port=7199
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dlog4j.configuration=log4j-server.properties
>> -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/dse.pid -cp
>> :/usr/share/dse/dse.jar:/usr/share/dse/common/commons-codec-1.6.jar:/usr/share/dse/common/commons-io-2.4.jar:/usr/share/dse/common/guava-13.0.jar:/usr/share/dse/common/jbcrypt-0.3m.jar:/usr/share/dse/common/log4j-1.2.16.jar:/usr/share/dse/common/slf4j-api-1.6.1.jar:/usr/share/dse/common/slf4j-log4j12-1.6.1.jar:/etc/dse:/usr/share/java/jna.jar:/etc/dse/cassandra:/usr/share/dse/cassandra/tools/lib/stress.jar:/usr/share/dse/cassandra/lib/antlr-2.7.7.jar:/usr/share/dse/cassandra/lib/antlr-3.2.jar:/usr/share/dse/cassandra/lib/antlr-runtime-3.2.jar:/usr/share/dse/cassandra/lib/avro-1.4.0-cassandra-1.jar:/usr/share/dse/cassandra/lib/cassandra-all-1.1.9.10.jar:/usr/share/dse/cassandra/lib/cassandra-clientutil-1.1.9.10.jar:/usr/share/dse/cassandra/lib/cassandra-thrift-1.1.9.10.jar:/usr/share/dse/cassandra/lib/commons-cli-1.1.jar:/usr/share/dse/cassandra/lib/commons-codec-1.6.jar:/usr/share/dse/cassandra/lib/commons-lang-2.4.jar:/usr/share/dse/cassandra/lib/commons-logging-1.1.1.jar:/usr/share/dse/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/dse/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/dse/cassandra/lib/guava-13.0.jar:/usr/share/dse/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/dse/cassandra/lib/httpclient-4.0.1.jar:/usr/share/dse/cassandra/lib/httpcore-4.0.1.jar:/usr/share/dse/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/dse/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/dse/cassandra/lib/jamm-0.2.5.jar:/usr/share/dse/cassandra/lib/jline-0.9.94.jar:/usr/share/dse/cassandra/lib/joda-time-1.6.2.jar:/usr/share/dse/cassandra/lib/json-simple-1.1.jar:/usr/share/dse/cassandra/lib/libthrift-0.7.0.jar:/usr/share/dse/cassandra/lib/log4j-1.2.16.jar:/usr/share/dse/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/dse/cassandra/lib/servlet-api-2.5.jar:/usr/share/dse/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/dse/cassandra/lib/snakeyaml-1.6.jar:/usr/share/dse/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/dse/cassandra/lib/snaptree-0.1.jar:/usr/share/dse/cassandra/lib/stringtemplate-3.2.jar::/usr/share/dse/solr/lib/solr-4.0.2.4-SNAPSHOT-uber.jar:/usr/share/dse/solr/lib/solr-web-4.0.2.4-SNAPSHOT.jar:/usr/share/dse/solr/conf::/usr/share/dse/tomcat/lib/annotations-api-6.0.32.jar:/usr/share/dse/tomcat/lib/catalina-6.0.32.jar:/usr/share/dse/tomcat/lib/catalina-ha-6.0.32.jar:/usr/share/dse/tomcat/lib/coyote-6.0.32.jar:/usr/share/dse/tomcat/lib/el-api-6.0.29.jar:/usr/share/dse/tomcat/lib/jasper-6.0.29.jar:/usr/share/dse/tomcat/lib/jasper-el-6.0.29.jar:/usr/share/dse/tomcat/lib/jasper-jdt-6.0.29.jar:/usr/share/dse/tomcat/lib/jsp-api-6.0.29.jar:/usr/share/dse/tomcat/lib/juli-6.0.32.jar:/usr/share/dse/tomcat/lib/servlet-api-6.0.29.jar:/usr/share/dse/tomcat/lib/tribes-6.0.32.jar:/usr/share/dse/tomcat/conf::/usr/share/dse/hadoop:/etc/dse/hadoop:/usr/share/dse/hadoop/lib/ant-1.6.5.jar:/usr/share/dse/hadoop/lib/automaton-1.11-8.jar:/usr/share/dse/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/share/dse/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/share/dse/hadoop/lib/commons-cli-1.2.jar:/usr/share/dse/hadoop/lib/commons-codec-1.4.jar:/usr/share/dse/hadoop/lib/commons-collections-3.2.1.jar:/usr/share/dse/hadoop/lib/commons-configuration-1.6.jar:/usr/share/dse/hadoop/lib/commons-digester-1.8.jar:/usr/share/dse/hadoop/lib/commons-el-1.0.jar:/usr/share/dse/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/share/dse/hadoop/lib/commons-lang-2.4.jar:/u
>>
>>
>>
>> Its the spike in RAM usage. Now it is normal but keeps showing the spikes.
>>
>>
>>
>> On Wed, Mar 26, 2014 at 5:31 PM, Marcin Cabaj <marcin.cabaj@datasift.com>
>> wrote:
>>
>> Hi,
>>
>>
>>
>> RSS or VIRT?
>>
>>
>>
>> Could you paste output of:
>> $ ps -p `jps | awk '/CassandraDaemon/ {print $1}'` uww
>>
>> please?
>>
>>
>>
>> On Wed, Mar 26, 2014 at 5:20 PM, prem yadav <ipremyadav@gmail.com> wrote:
>>
>> Hi,
>>
>> in another thread, I has mentioned that we had issue with Cassandra
>> getting killed by kernel due to OOM. Downgrading to jdk6_37 seems to have
>> fixed it.
>>
>>
>>
>> However, even now, after every couple of hours, the nodes are showing a
>> spike in memory usage.
>>
>> For ex: on a 8GB ram machine, once the usage reached to 7.5 GB.
>>
>> and then slowly it comes down to normal.
>>
>>
>>
>> Cassandra version in use is 1.1.9.10.
>>
>>
>>
>> Any idea why this could be happening? There is no load on the cluster.
>>
>>
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>

Mime
View raw message