Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 99165 invoked from network); 19 Dec 2010 21:39:46 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 19 Dec 2010 21:39:46 -0000 Received: (qmail 58411 invoked by uid 500); 19 Dec 2010 21:39:44 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 58391 invoked by uid 500); 19 Dec 2010 21:39:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 58383 invoked by uid 99); 19 Dec 2010 21:39:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Dec 2010 21:39:44 +0000 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of victor.kabdebon@gmail.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-iw0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Dec 2010 21:39:38 +0000 Received: by iwn40 with SMTP id 40so2746574iwn.31 for ; Sun, 19 Dec 2010 13:39:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type; bh=dsYWzrS/8iqr3QuK5JVsyfn68B1hdYmk1DXwhksbFVM=; b=Pj5oKjWDpiVklVESFXIBXzNIfzlwwu1GPXCxozdwT2SfJ91d9H7TLAhGVH8V1kQK8c JsRyjw/RatwUwIL50llX93EPyd+oc7vnGys27Kp7SyPIPuG2o2D8zTvFprbSgCVKSiB/ P6tCUAdzb13Dgw+6EJ4E9uyNkO34y4tfA9lc4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=P8C3oYdYHDIycUlusqbRHIZF3Qiih/0SNsJf5KAuiwSegiBfXQUkZ7Dufzj95VbMlL mM4Vd+TurDxpjILeL2OYANgHMRWZa0jK/KrQ6jiJJQtMlVb81EX0UrsmaF6wL3nknchr XdG/7qYt3UwqFCZkwp8iUmPovyd0Na4FhXrng= Received: by 10.42.177.6 with SMTP id bg6mr3432232icb.79.1292794755958; Sun, 19 Dec 2010 13:39:15 -0800 (PST) MIME-Version: 1.0 Received: by 10.42.222.10 with HTTP; Sun, 19 Dec 2010 13:38:55 -0800 (PST) In-Reply-To: References: <3CB61C05-8FEE-4C88-B179-EBBD488E77A1@backupify.com> From: Victor Kabdebon Date: Sun, 19 Dec 2010 16:38:55 -0500 Message-ID: Subject: Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables) To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=90e6ba6e89ee32465d0497ca3cda X-Virus-Checked: Checked by ClamAV on apache.org --90e6ba6e89ee32465d0497ca3cda Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello Peter, So more information on that problem : Yes I am using this node with very few data, it is used to design requests so I don't need a very large dataset. I am running Apache Cassandra 0.6.6 on a Debian Stable, with java version "1.6.0_22". I recently restarted cassandra, thus I have this low memory use, but if I keep it running for 2 or 3 weeks then Cassandra will take about 1.5 Gb. Her= e is the result of the command, one day after the previous one : victor@***:~$ sudo ps aux | grep "cassandra" root 11034 0.2 26.8 1167304* 540176* ? Sl Dec17 8:09 /usr/bin/java -ea -Xms128M -Xmx512M -XX:+UseParNewGC -XX:+UseConcMarkSweepG= C -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=3D8 -XX:MaxTenuringThreshol= d=3D1 -XX:CMSInitiatingOccupancyFraction=3D75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.port=3D8081 -Dcom.sun.management.jmxremote.ssl=3Dfalse -Dcom.sun.management.jmxremote.authenticate=3Dfalse -Dstorage-config=3Dbin/../conf -Dcassandra-foreground=3Dyes -cp bin/../conf:bin/../build/classes:bin/../lib/antlr-3.1.3.jar:bin/../lib/apac= he-cassandra-0.6.6.jar:bin/../lib/clhm-production.jar:bin/../lib/commons-cl= i-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-collections-3= .2.1.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/google-collections-1.0.= jar:bin/../lib/hadoop-core-0.20.1.jar:bin/../lib/high-scale-lib.jar:bin/../= lib/ivy-2.1.0.jar:bin/../lib/jackson-core-asl-1.4.0.jar:bin/../lib/jackson-= mapper-asl-1.4.0.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/json-simple-1.1= .jar:bin/../lib/libthrift-r917130.jar:bin/../lib/log4j-1.2.14.jar:bin/../li= b/slf4j-api-1.5.8.jar:bin/../lib/slf4j-log4j12-1.5.8.jar org.apache.cassandra.thrift.CassandraDaemon I have done very little work on it (a few insert and reads). Thank you, Victor 2010/12/19 Peter Schuller > > victor@****:~$ sudo ps aux | grep "cassandra" > > cassandra 11034 0.2 22.9 1107772 462764 ? Sl Dec17 6:13 > > /usr/bin/java -ea -Xms128M -Xmx512M -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC > > -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=3D8 > -XX:MaxTenuringThreshold=3D1 > > -XX:CMSInitiatingOccupancyFraction=3D75 -XX:+UseCMSInitiatingOccupancyO= nly > > -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.port=3D8= 081 > > -Dcom.sun.management.jmxremote.ssl=3Dfalse > > -Dcom.sun.management.jmxremote.authenticate=3Dfalse > > -Dstorage-config=3Dbin/../conf -Dcassandra-foreground=3Dyes -cp > > > bin/../conf:bin/../build/classes:bin/../lib/antlr-3.1.3.jar:bin/../lib/ap= ache-cassandra-0.6.6.jar:bin/../lib/clhm-production.jar:bin/../lib/commons-= cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-collections= -3.2.1.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/google-collections-1.= 0.jar:bin/../lib/hadoop-core-0.20.1.jar:bin/../lib/high-scale-lib.jar:bin/.= ./lib/ivy-2.1.0.jar:bin/../lib/jackson-core-asl-1.4.0.jar:bin/../lib/jackso= n-mapper-asl-1.4.0.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/json-simple-1= .1.jar:bin/../lib/libthrift-r917130.jar:bin/../lib/log4j-1.2.14.jar:bin/../= lib/slf4j-api-1.5.8.jar:bin/../lib/slf4j-log4j12-1.5.8.jar > > org.apache.cassandra.thrift.CassandraDaemon > > > > Cassandra uses 462764 Kb, roughly 460 Mb for 2 Mb of data... And it kee= ps > > getting bigger. > > It is important to know that I have just a few insert, quite a lot of > read > > though. Also Cassandra seams to completly ignore the JVM limitations su= ch > as > > Xmx. > > If I don't stop and launch Cassandra every 15 ou 20 days it simply > crashes, > > due to oom errors. > > The resident size is not unexpected given that your Xmx is 512 MB. The > virtual may or may not be expected depending; for example thread > stacks as previously discussed in this thread. > > If you're not seeing the *resident* set size go above the maximum heap > size, you're unlikely to be seeing the same problem. > > WIth respect to OOM, see > http://www.riptano.com/docs/0.6/operations/tuning - but without more > information it's difficult to know what specifically it is that you're > hitting. Are you seriously saying you're running for 15-20 days with > only 2 mb of live data? > > -- > / Peter Schuller > --90e6ba6e89ee32465d0497ca3cda Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello Peter,

So more information on that problem :
Yes I am using= this node with very few data, it is used to design requests so I don't= need a very large dataset.
I am running Apache Cassandra 0.6.6 on a Deb= ian Stable, with java version "1.6.0_22".

I recently restarted cassandra, thus I have this low memory use, but if= I keep it running for 2 or 3 weeks then Cassandra will take about 1.5 Gb. = Here is the result of the command, one day after the previous one :

victor@***:~$ sudo ps aux | grep "cassandra"

root=A0= =A0=A0=A0 11034=A0 0.2 26.8 1167304 540176 ?=A0=A0=A0=A0=A0 Sl=A0=A0= Dec17=A0=A0 8:09 /usr/bin/java -ea -Xms128M -Xmx512M -XX:+UseParNewGC -XX:= +UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=3D8 -XX= :MaxTenuringThreshold=3D1 -XX:CMSInitiatingOccupancyFraction=3D75 -XX:+UseC= MSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.managem= ent.jmxremote.port=3D8081 -Dcom.sun.management.jmxremote.ssl=3Dfalse -Dcom.= sun.management.jmxremote.authenticate=3Dfalse -Dstorage-config=3Dbin/../con= f -Dcassandra-foreground=3Dyes -cp bin/../conf:bin/../build/classes:bin/../= lib/antlr-3.1.3.jar:bin/../lib/apache-cassandra-0.6.6.jar:bin/../lib/clhm-p= roduction.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.j= ar:bin/../lib/commons-collections-3.2.1.jar:bin/../lib/commons-lang-2.4.jar= :bin/../lib/google-collections-1.0.jar:bin/../lib/hadoop-core-0.20.1.jar:bi= n/../lib/high-scale-lib.jar:bin/../lib/ivy-2.1.0.jar:bin/../lib/jackson-cor= e-asl-1.4.0.jar:bin/../lib/jackson-mapper-asl-1.4.0.jar:bin/../lib/jline-0.= 9.94.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-r917130.jar:bi= n/../lib/log4j-1.2.14.jar:bin/../lib/slf4j-api-1.5.8.jar:bin/../lib/slf4j-l= og4j12-1.5.8.jar org.apache.cassandra.thrift.CassandraDaemon

I have done very little work on it (a few insert and reads).

Tha= nk you,
Victor

2010/12/19 Peter Schull= er <peter.schuller@infidyne.com>
> victor@= ****:~$ sudo ps aux | grep "cassandra"
> cassandra =A0=A0=A0 11034=A0 0.2 22.9 1107772 462764 ?=A0=A0=A0=A0=A0 = Sl=A0=A0 Dec17=A0=A0 6:13
> /usr/bin/java -ea -Xms128M -Xmx512M -XX:+UseParNewGC -XX:+UseConcMarkS= weepGC
> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=3D8 -XX:MaxTenuringThr= eshold=3D1
> -XX:CMSInitiatingOccupancyFraction=3D75 -XX:+UseCMSInitiatingOccupancy= Only
> -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.port=3D= 8081
> -Dcom.sun.management.jmxremote.ssl=3Dfalse
> -Dcom.sun.management.jmxremote.authenticate=3Dfalse
> -Dstorage-config=3Dbin/../conf -Dcassandra-foreground=3Dyes -cp
> bin/../conf:bin/../build/classes:bin/../lib/antlr-3.1.3.jar:bin/../lib= /apache-cassandra-0.6.6.jar:bin/../lib/clhm-production.jar:bin/../lib/commo= ns-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-collecti= ons-3.2.1.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/google-collections= -1.0.jar:bin/../lib/hadoop-core-0.20.1.jar:bin/../lib/high-scale-lib.jar:bi= n/../lib/ivy-2.1.0.jar:bin/../lib/jackson-core-asl-1.4.0.jar:bin/../lib/jac= kson-mapper-asl-1.4.0.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/json-simpl= e-1.1.jar:bin/../lib/libthrift-r917130.jar:bin/../lib/log4j-1.2.14.jar:bin/= ../lib/slf4j-api-1.5.8.jar:bin/../lib/slf4j-log4j12-1.5.8.jar
> org.apache.cassandra.thrift.CassandraDaemon
>
> Cassandra uses 462764 Kb, roughly 460 Mb for 2 Mb of data... And it ke= eps
> getting bigger.
> It is important to know that I have just a few insert, quite a lot of = read
> though. Also Cassandra seams to completly ignore the JVM limitations s= uch as
> Xmx.
> If I don't stop and launch Cassandra every 15 ou 20 days it simply= crashes,
> due to oom errors.

The resident size is not unexpected given that your Xmx is 512 MB. Th= e
virtual may or may not be expected depending; for example thread
stacks as previously discussed in this thread.

If you're not seeing the *resident* set size go above the maximum heap<= br> size, you're unlikely to be seeing the same problem.

WIth respect to OOM, see
http://www.riptano.com/docs/0.6/operations/tuning - but without mo= re
information it's difficult to know what specifically it is that you'= ;re
hitting. Are you seriously saying you're running for 15-20 days with only 2 mb of live data?

--
/ Peter Schuller

--90e6ba6e89ee32465d0497ca3cda--