cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jeff.ji...@crowdstrike.com>
Subject Re: Too High resident memory of cassandra 2.2.8
Date Mon, 14 Nov 2016 13:45:20 GMT
nodetool cfstats will show it per table.

 

The bloom filter / compression data is typically (unless you have very unusual settings in
your schema) 1-3GB each per TB of data, so with 235’ish GB/server, it’s unlikely bloom
filter or compression data.

 

The memTable is AT LEAST 1MB per columnfamily/table, so if you know how many tables you have,
that may be an initial lower bound guess.

 

 

 

From: ankit tyagi <ankittyagi.mnnit@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Sunday, November 13, 2016 at 11:33 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Too High resident memory of cassandra 2.2.8

 

Hi Jeff,

Below is the output of nodetool staus command.

 

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address         Load       Tokens       Owns    Host ID                              
Rack

UN  192.168.68.156  235.79 GB  256          ?       e7b1a44d-0cd2-4b60-b322-4f989933fc51 
rack1

UN  192.168.68.157  234.65 GB  256          ?       70406f0b-3620-401e-beaa-15deb4b799ce 
rack1

UN  192.168.69.146   256          ?       d32e1e4d-ec86-4c3f-9397-11f37ff7b4d3  rack1

UN  192.168.69.147  242.77 GB  256          ?       646d9416-a467-4526-9656-959aa98404d0 
rack1

UN  192.168.69.148  249.84 GB  256          ?       9b0ab632-75f4-4781-a987-a00b8246ae97 
rack1

UN  192.168.69.149  240.62 GB  256          ?       406c4d3e-0933-4cba-935f-bfba16e6d878 
rack1

 

 

is there any command to find out the size of offheap memtable. 

 

On Mon, Nov 14, 2016 at 12:30 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:

Cassandra keeps certain data structures offheap, including bloom filters (scales with data
size), compression metadata (scales with data size), and potentially memtables (scales with
# of keyspaces/tables).

 

How much data on your node? Onheap or offheap memtables?

 

 

 

From: ankit tyagi <ankittyagi.mnnit@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Sunday, November 13, 2016 at 10:55 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Too High resident memory of cassandra 2.2.8

 

Hi, 

 

we are using cassandra 2.2.8 version in production. we are seeing resident memory of cassndra
process is very high 40G while heap size is only 8GB.

 

root      23339      1 80 Nov11 ?        2-09:38:08 /opt/java8/bin/java -ea -javaagent:bin/../lib/jamm-0.3.0.jar
-XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M
-Xmx8192M -Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+PerfDisableSharedMem
-XX:CompileCommandFile=bin/../conf/hotspot_compiler -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled
-XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=10000 -XX:+UseCondCardMark -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure -Xloggc:bin/../logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=10M -Djava.net.preferIPv4Stack=true -Dcassandra.jmx.local.port=7199 -XX:+DisableExplicitGC
-Djava.library.path=bin/../lib/sigar-bin -javaagent:/myntra/currentCassandra/lib/agent-1.2.jar=statsd.myntra.com:8125
-Dlogback.configurationFile=logback.xml -Dcassandra.logdir=bin/../logs -Dcassandra.storagedir=bin/../data
-cp bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/agent-1.2.jar:bin/../lib/airline-0.6.jar:bin/../lib/antlr-runtime-3.5.2.jar:bin/../lib/apache-cassandra-2.2.8.jar:bin/../lib/apache-cassandra-clientutil-2.2.8.jar:bin/../lib/apache-cassandra-thrift-2.2.8.jar:bin/../lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang3-3.1.jar:bin/../lib/commons-math3-3.2.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:bin/../lib/crc32ex-0.1.1.jar:bin/../lib/disruptor-3.0.1.jar:bin/../lib/ecj-4.4.2.jar:bin/../lib/guava-16.0.jar:bin/../lib/high-scale-lib-1.0.6.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.3.0.jar:bin/../lib/java-statsd-client-3.1.0.jar:bin/../lib/javax.inject.jar:bin/../lib/jbcrypt-0.3m.jar:bin/../lib/jcl-over-slf4j-1.7.7.jar:bin/../lib/jna-4.0.0.jar:bin/../lib/joda-time-2.4.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.9.2.jar:bin/../lib/log4j-over-slf4j-1.7.7.jar:bin/../lib/logback-classic-1.1.3.jar:bin/../lib/logback-core-1.1.3.jar:bin/../lib/lz4-1.3.0.jar:bin/../lib/metrics-core-2.2.0.jar:bin/../lib/metrics-core-3.1.0.jar:bin/../lib/metrics-jvm-3.1.0.jar:bin/../lib/metrics-logback-3.1.0.jar:bin/../lib/metrics-statsd-2.3.0.jar:bin/../lib/netty-all-4.0.23.Final.jar:bin/../lib/ohc-core-0.3.4.jar:bin/../lib/ohc-core-j8-0.3.4.jar:bin/../lib/reporter-config3-3.0.0.jar:bin/../lib/reporter-config-base-3.0.0.jar:bin/../lib/sigar-1.6.4.jar:bin/../lib/slf4j-api-1.7.7.jar:bin/../lib/snakeyaml-1.11.jar:bin/../lib/snappy-java-1.1.1.7.jar:bin/../lib/ST4-4.0.8.jar:bin/../lib/stream-2.5.2.jar:bin/../lib/super-csv-2.1.0.jar:bin/../lib/thrift-server-0.3.7.jar:bin/../lib/jsr223/*/*.jar
org.apache.cassandra.service.CassandraDaemon

 

 

below is the output of top command.

  

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

 23339 root      20   0  295g  39g  28g S 107.5 63.5   3458:31 java

 

 

I couldn't understand why  cassandra process is using so much resident memory while heap size
is only 8gb. Any insight??

 


Mime
View raw message