Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D9F26200BBF for ; Mon, 14 Nov 2016 08:34:06 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id D86CB160B06; Mon, 14 Nov 2016 07:34:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id ADFE9160B05 for ; Mon, 14 Nov 2016 08:34:05 +0100 (CET) Received: (qmail 48102 invoked by uid 500); 14 Nov 2016 07:34:04 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 48092 invoked by uid 99); 14 Nov 2016 07:34:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Nov 2016 07:34:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 9A51A1A79FC for ; Mon, 14 Nov 2016 07:34:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.079 X-Spam-Level: *** X-Spam-Status: No, score=3.079 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_LINEPADDING=1.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Oa4oQkPMx5pE for ; Mon, 14 Nov 2016 07:34:01 +0000 (UTC) Received: from mail-yw0-f175.google.com (mail-yw0-f175.google.com [209.85.161.175]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id CDA9B5FC5D for ; Mon, 14 Nov 2016 07:34:00 +0000 (UTC) Received: by mail-yw0-f175.google.com with SMTP id t125so49934576ywc.1 for ; Sun, 13 Nov 2016 23:34:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=+hyTywJTGhSxyRWPFfGFHypu+2+k1O/rEzimEEnMm8I=; b=vhEli5h9S/iOFTaxLo6Za2ZgD6QJUdPKhYJqpqD7NF5dbB62acJKxmJvNZQNUVImVx SaNc4Tg9jc/O4ZY53KdwCpOrF31AkJQNUnfEh7JYfKrBjPULYtDa0+Wq1ib9CDuBRohH Uf+jvVujzOLprFBc2ioBOA2Fl62iuNoqzNxw0A4WoZ4mIlvXBdR+h0VFbugWJRVTUzoZ DHjUjKBMp34qt0oK/YwHt7EhOq0SSk4dhunElNZZ5bs5itS2vWZ1oEEh5PJ9ngd6UlXe ohdQwd2+yskEzg0F4VFLNp2P7F2Eq0BKy52JoCVoYBhMxMGVG+FLaOrN7jK+eCtRHLMg mj4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=+hyTywJTGhSxyRWPFfGFHypu+2+k1O/rEzimEEnMm8I=; b=flnMz1/y5oOmpJlQZZznzj8zKYbQf0dQhv+LfqrMel2PRoxGU1ElGsPA9TNkElLlnA wfg5minI/xnzqlErF27EOY2YC3TXZA/MbbFp28ADEyKfylSD86XotXXxeiMJrf8OTPQ/ xSaBSkEplot4g0Yta4XVtG+/hSzOAGcGZA5IwqrpL8vNolwiS/lMVSf2+I4LSJrsbVgq ixhpklRXzv4LiooaR4AOAkh+hyVt7i11WdC+EttTM7yaXSRIlgTwH+W+MFXsq17QqjIj f0R6tCVN8l0FJ5wwQtF0e3h7qbwmVyqo1pXcXhRlZlWZ0SrAvCKTGSaZvCjQuHxuefkY b2Sw== X-Gm-Message-State: ABUngvfJ6FRYDsxC69fXzHUDa4Xkaylun9Fo1nI09QGq9eghACl7pVro9edRiXqwRhrqkfjtSjWfY7Mywr63ow== X-Received: by 10.157.8.242 with SMTP id 105mr6010051otf.32.1479108840296; Sun, 13 Nov 2016 23:34:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.202.88.69 with HTTP; Sun, 13 Nov 2016 23:33:59 -0800 (PST) In-Reply-To: References: From: ankit tyagi Date: Mon, 14 Nov 2016 13:03:59 +0530 Message-ID: Subject: Re: Too High resident memory of cassandra 2.2.8 To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113596c80294d405413dddb1 archived-at: Mon, 14 Nov 2016 07:34:07 -0000 --001a113596c80294d405413dddb1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Jeff, Below is the output of nodetool staus command. Status=3DUp/Down |/ State=3DNormal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 192.168.68.156 *235.79 GB* 256 ? e7b1a44d-0cd2-4b60-b322-4f989933fc51 rack1 UN 192.168.68.157 *234.65 GB* 256 ? 70406f0b-3620-401e-beaa-15deb4b799ce rack1 UN 192.168.69.146 256 ? d32e1e4d-ec86-4c3f-9397-11f37ff7b4d3 rack1 UN 192.168.69.147 *242.77 GB * 256 ? 646d9416-a467-4526-9656-959aa98404d0 rack1 UN 192.168.69.148 *249.84 GB * 256 ? 9b0ab632-75f4-4781-a987-a00b8246ae97 rack1 UN 192.168.69.149 *240.62 GB* 256 ? 406c4d3e-0933-4cba-935f-bfba16e6d878 rack1 is there any command to find out the size of offheap memtable. On Mon, Nov 14, 2016 at 12:30 PM, Jeff Jirsa wrote: > Cassandra keeps certain data structures offheap, including bloom filters > (scales with data size), compression metadata (scales with data size), an= d > potentially memtables (scales with # of keyspaces/tables). > > > > How much data on your node? Onheap or offheap memtables? > > > > > > > > *From: *ankit tyagi > *Reply-To: *"user@cassandra.apache.org" > *Date: *Sunday, November 13, 2016 at 10:55 PM > *To: *"user@cassandra.apache.org" > *Subject: *Too High resident memory of cassandra 2.2.8 > > > > Hi, > > > > we are using cassandra 2.2.8 version in production. we are seeing residen= t > memory of cassndra process is very high 40G while heap size is only 8GB. > > > > root 23339 1 80 Nov11 ? 2-09:38:08 /opt/java8/bin/java > -ea -javaagent:bin/../lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled > -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=3D42* -Xms8192M -Xmx819= 2M > -Xmn2048M* -XX:+HeapDumpOnOutOfMemoryError -Xss256k > -XX:StringTableSize=3D1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=3D8 > -XX:MaxTenuringThreshold=3D1 -XX:CMSInitiatingOccupancyFraction=3D75 -XX:= +UseCMSInitiatingOccupancyOnly > -XX:+UseTLAB -XX:+PerfDisableSharedMem -XX:CompileCommandFile=3Dbin/../co= nf/hotspot_compiler > -XX:CMSWaitDuration=3D10000 -XX:+CMSParallelInitialMarkEnabled > -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=3D10000 > -XX:+UseCondCardMark -XX:+PrintGCDetails -XX:+PrintGCDateStamps > -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplication= StoppedTime > -XX:+PrintPromotionFailure -Xloggc:bin/../logs/gc.log > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3D10 -XX:GCLogFileSize= =3D10M > -Djava.net.preferIPv4Stack=3Dtrue -Dcassandra.jmx.local.port=3D7199 > -XX:+DisableExplicitGC -Djava.library.path=3Dbin/../lib/sigar-bin > -javaagent:/myntra/currentCassandra/lib/agent-1.2.jar=3D > statsd.myntra.com:8125 > > -Dlogback.configurationFile=3Dlogback.xml -Dcassandra.logdir=3Dbin/../log= s > -Dcassandra.storagedir=3Dbin/../data -cp bin/../conf:bin/../build/ > classes/main:bin/../build/classes/thrift:bin/../lib/ > agent-1.2.jar:bin/../lib/airline-0.6.jar:bin/../lib/ > antlr-runtime-3.5.2.jar:bin/../lib/apache-cassandra-2.2.8. > jar:bin/../lib/apache-cassandra-clientutil-2.2.8.jar:bin/../lib/apache- > cassandra-thrift-2.2.8.jar:bin/../lib/cassandra-driver- > core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:bin/../ > lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar: > bin/../lib/commons-lang3-3.1.jar:bin/../lib/commons-math3- > 3.2.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/ > concurrentlinkedhashmap-lru-1.4.jar:bin/../lib/crc32ex-0.1. > 1.jar:bin/../lib/disruptor-3.0.1.jar:bin/../lib/ecj-4.4.2. > jar:bin/../lib/guava-16.0.jar:bin/../lib/high-scale-lib-1.0. > 6.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/ > jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.3.0.jar:bin/ > ../lib/java-statsd-client-3.1.0.jar:bin/../lib/javax.inject. > jar:bin/../lib/jbcrypt-0.3m.jar:bin/../lib/jcl-over-slf4j- > 1.7.7.jar:bin/../lib/jna-4.0.0.jar:bin/../lib/joda-time-2. > 4.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift- > 0.9.2.jar:bin/../lib/log4j-over-slf4j-1.7.7.jar:bin/../ > lib/logback-classic-1.1.3.jar:bin/../lib/logback-core-1.1.3. > jar:bin/../lib/lz4-1.3.0.jar:bin/../lib/metrics-core-2.2.0. > jar:bin/../lib/metrics-core-3.1.0.jar:bin/../lib/metrics- > jvm-3.1.0.jar:bin/../lib/metrics-logback-3.1.0.jar:bin/ > ../lib/metrics-statsd-2.3.0.jar:bin/../lib/netty-all-4.0. > 23.Final.jar:bin/../lib/ohc-core-0.3.4.jar:bin/../lib/ohc- > core-j8-0.3.4.jar:bin/../lib/reporter-config3-3.0.0.jar: > bin/../lib/reporter-config-base-3.0.0.jar:bin/../lib/ > sigar-1.6.4.jar:bin/../lib/slf4j-api-1.7.7.jar:bin/../ > lib/snakeyaml-1.11.jar:bin/../lib/snappy-java-1.1.1.7.jar: > bin/../lib/ST4-4.0.8.jar:bin/../lib/stream-2.5.2.jar:bin/../ > lib/super-csv-2.1.0.jar:bin/../lib/thrift-server-0.3.7.jar:bin/../lib/jsr= 223/*/*.jar > org.apache.cassandra.service.CassandraDaemon > > > > > > below is the output of top command. > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 23339 root 20 0 295g * 39g 28g* S 107.5 63.5 3458:31 java > > > > > > *I couldn't understand why cassandra process is using so much resident > memory while heap size is only 8gb. Any insight??* > --001a113596c80294d405413dddb1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Jeff,
Below is the output of nodetool st= aus command.

Status=3DUp/Down
|/ State= =3DNormal/Leaving/Joining/Moving
-- =C2=A0Address =C2=A0 =C2=A0 = =C2=A0 =C2=A0 Load =C2=A0 =C2=A0 =C2=A0 Tokens =C2=A0 =C2=A0 =C2=A0 Owns = =C2=A0 =C2=A0Host ID =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Rack
UN =C2= =A0192.168.68.156 =C2=A0235.79 GB =C2=A0256 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0? =C2=A0 =C2=A0 =C2=A0 e7b1a44d-0cd2-4b60-b322-4f989933fc51 =C2= =A0rack1
UN =C2=A0192.168.68.157 =C2=A0234.65 GB =C2=A0256= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0? =C2=A0 =C2=A0 =C2=A0 70406f0b-3620-401= e-beaa-15deb4b799ce =C2=A0rack1
UN =C2=A0192.168.69.146 =C2=A0 25= 6 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0? =C2=A0 =C2=A0 =C2=A0 d32e1e4d-ec86-4c= 3f-9397-11f37ff7b4d3 =C2=A0rack1
UN =C2=A0192.168.69.147 =C2=A0242.77 GB =C2=A0256 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0? =C2=A0 =C2=A0 = =C2=A0 646d9416-a467-4526-9656-959aa98404d0 =C2=A0rack1
UN =C2=A0= 192.168.69.148 =C2=A0249.84 GB =C2=A0256 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0? =C2=A0 =C2=A0 =C2=A0 9b0ab632-75f4-4781-a987-a00b8246ae97 =C2=A0ra= ck1
UN =C2=A0192.168.69.149 =C2=A0240.62 GB =C2=A0256 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0? =C2=A0 =C2=A0 =C2=A0 406c4d3e-0933-4cba-93= 5f-bfba16e6d878 =C2=A0rack1


is ther= e any command to find out the size of offheap memtable.=C2=A0

On Mon, Nov 14, 201= 6 at 12:30 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:

Cassandra keeps certain data structures offheap, including bl= oom filters (scales with data size), compression metadata (scales with data= size), and potentially memtables (scales with # of keyspaces/tables).

=C2=A0

= How much data on your = node? Onheap or offheap memtables?

=C2=A0

=C2=A0

=C2=A0

From: ankit tyagi <ankittyagi.mnnit@gmail.com>
Reply-To: "user@cassandra.apa= che.org" <user@cassandra.apache.org>
Date: Sunday, Novem= ber 13, 2016 at 10:55 PM
To: "user@cassandra.apache.org" <user@cassandra= .apache.org>
Subject: Too High resident memory of cassandr= a 2.2.8

=C2=A0

Hi,

=C2=A0<= /u>

we are using cassandra 2.2.8 versi= on in production. we are seeing resident memory of cassndra process is very= high 40G while heap size is only 8GB.

=C2=A0

ro= ot =C2=A0 =C2=A0 =C2=A023339 =C2=A0 =C2=A0 =C2=A01 80 Nov11 ? =C2=A0 =C2=A0= =C2=A0 =C2=A02-09:38:08 /opt/java8/bin/java -ea -javaagent:bin/../lib/jamm= -0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:= ThreadPriorityPolicy=3D42 -Xms8192M -Xmx8192M -Xmn2048M -XX:+He= apDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=3D1000003 -XX:+UsePar= NewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRat= io=3D8 -XX:MaxTenuringThreshold=3D1 -XX:CMSInitiatingOccupancyFraction= =3D75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+PerfDi= sableSharedMem -XX:CompileCommandFile=3Dbin/../conf/hotspot_compiler -= XX:CMSWaitDuration=3D10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMS= EdenChunksRecordAlways -XX:CMSWaitDuration=3D10000 -XX:+UseCondCardMark -XX= :+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuri= ngDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionF= ailure -Xloggc:bin/../logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCL= ogFiles=3D10 -XX:GCLogFileSize=3D10M -Djava.net.preferIPv4Stack=3Dtrue= -Dcassandra.jmx.local.port=3D7199 -XX:+DisableExplicitGC -Djava.libra= ry.path=3Dbin/../lib/sigar-bin -javaagent:/myntra/currentCassandr= a/lib/agent-1.2.jar=3Dstatsd.= myntra.com:8125 -Dlogback.configurationFile=3Dlogback.xml -Dcassan= dra.logdir=3Dbin/../logs -Dcassandra.storagedir=3Dbin/../data -cp bin/= ../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin= /../lib/agent-1.2.jar:bin/../lib/airline-0.6.jar:bin/../lib/= antlr-runtime-3.5.2.jar:bin/../lib/apache-cassandra-2.2.8.jar:bin= /../lib/apache-cassandra-clientutil-2.2.8.jar:bin/../lib/apache-<= wbr>cassandra-thrift-2.2.8.jar:bin/../lib/cassandra-driver-core-2= .2.0-rc2-SNAPSHOT-20150617-shaded.jar:bin/../lib/commons-cli-1.1.= jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang3-3.1= .jar:bin/../lib/commons-math3-3.2.jar:bin/../lib/compress-lz= f-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:bin/..= /lib/crc32ex-0.1.1.jar:bin/../lib/disruptor-3.0.1.jar:bin/../lib/= ecj-4.4.2.jar:bin/../lib/guava-16.0.jar:bin/../lib/high-scale-lib= -1.0.6.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/= jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.3.0.jar:bin/../lib= /java-statsd-client-3.1.0.jar:bin/../lib/javax.inject.jar:bin/../= lib/jbcrypt-0.3m.jar:bin/../lib/jcl-over-slf4j-1.7.7.jar:bin/../l= ib/jna-4.0.0.jar:bin/../lib/joda-time-2.4.jar:bin/../lib/json-sim= ple-1.1.jar:bin/../lib/libthrift-0.9.2.jar:bin/../lib/log4j-= over-slf4j-1.7.7.jar:bin/../lib/logback-classic-1.1.3.jar:bin/../= lib/logback-core-1.1.3.jar:bin/../lib/lz4-1.3.0.jar:bin/../lib/me= trics-core-2.2.0.jar:bin/../lib/metrics-core-3.1.0.jar:bin/../lib= /metrics-jvm-3.1.0.jar:bin/../lib/metrics-logback-3.1.0.jar:bin/<= wbr>../lib/metrics-statsd-2.3.0.jar:bin/../lib/netty-all-4.0.23.F= inal.jar:bin/../lib/ohc-core-0.3.4.jar:bin/../lib/ohc-core-j8-0.3= .4.jar:bin/../lib/reporter-config3-3.0.0.jar:bin/../lib/reporter-= config-base-3.0.0.jar:bin/../lib/sigar-1.6.4.jar:bin/../lib/= slf4j-api-1.7.7.jar:bin/../lib/snakeyaml-1.11.jar:bin/../lib/snap= py-java-1.1.1.7.jar:bin/../lib/ST4-4.0.8.jar:bin/../lib/stream-2.= 5.2.jar:bin/../lib/super-csv-2.1.0.jar:bin/../lib/thrift-server-0= .3.7.jar:bin/../lib/jsr223/*/*.jar org.apache.cassandra.service.C= assandraDaemon

= =C2=A0

=C2=A0

=

below is the output of top command.<= u>

=C2=A0=C2=A0

PID USER =C2=A0 =C2=A0 =C2=A0PR =C2= =A0NI =C2=A0VIRT =C2=A0RES =C2=A0SHR S %CPU %MEM =C2=A0 =C2=A0TIME+ =C2=A0C= OMMAND

=C2=A023339 root = =C2=A0 =C2=A0 =C2=A020 =C2=A0 0 =C2=A0295g =C2=A039g =C2=A028g S 107= .5 63.5 =C2=A0 3458:31 java

=C2=A0

= =C2=A0

I couldn't unders= tand why =C2=A0cassandra process is using so much resident memory while hea= p size is only 8gb. Any insight??


--001a113596c80294d405413dddb1--