Return-Path: Delivered-To: apmail-cassandra-dev-archive@www.apache.org Received: (qmail 20584 invoked from network); 16 Dec 2010 07:40:17 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 16 Dec 2010 07:40:17 -0000 Received: (qmail 48836 invoked by uid 500); 16 Dec 2010 07:40:14 -0000 Delivered-To: apmail-cassandra-dev-archive@cassandra.apache.org Received: (qmail 48731 invoked by uid 500); 16 Dec 2010 07:40:13 -0000 Mailing-List: contact dev-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list dev@cassandra.apache.org Received: (qmail 48636 invoked by uid 99); 16 Dec 2010 07:40:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 07:40:13 +0000 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,HTML_OBFUSCATE_05_10,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of schumi.han@gmail.com designates 209.85.161.43 as permitted sender) Received: from [209.85.161.43] (HELO mail-fx0-f43.google.com) (209.85.161.43) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 07:39:42 +0000 Received: by fxm18 with SMTP id 18so3092706fxm.30 for ; Wed, 15 Dec 2010 23:39:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=/OzybLxvhdQIoqyPO7/7Ar2b2k4xXTFiHRIiWIqpGCA=; b=ZKltoyrHgFZXIYBTv3b/+8sT9vJ6Vmm0zF/H5taG5BDWuBiGEcIX3rcu1yB1RDL3D2 g+Kyh9QNA+O/orLeKBUBkgNHkATrw8Z/6IvNVofi1FqQGdFFaVdp1npUpOK7jkKoIjKj XM80DGVpEacHG4/169E7XXcLTeoBWGsVVCTDg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=llIiuSS0pzy1+6j5eYDNREFiXWc0knWSJl+NW0jfu9XZvPV0sqwPDbNCumUGGOEeMb mIEobdtpXThoVJ9kKK1cDtoQWtMmdL8gbd5bGL2wdgkKJifIgiaY0At44b8eJIMga8GZ ZH1+rkT0EYm4pfeHYp6wu10lFq8hjWZ+6QqFI= MIME-Version: 1.0 Received: by 10.223.114.14 with SMTP id c14mr38241faq.103.1292485160724; Wed, 15 Dec 2010 23:39:20 -0800 (PST) Received: by 10.223.96.130 with HTTP; Wed, 15 Dec 2010 23:39:20 -0800 (PST) In-Reply-To: References:

Date: Thu, 16 Dec 2010 15:39:20 +0800 Message-ID: Subject: Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables) From: Zhu Han To: user@cassandra.apache.org Cc: dev@cassandra.apache.org Content-Type: multipart/alternative; boundary=001636c598dbe1eaa9049782260c X-Virus-Checked: Checked by ClamAV on apache.org --001636c598dbe1eaa9049782260c Content-Type: text/plain; charset=ISO-8859-1 Sorry for spam again. :-) I think I find the root cause. Here is a bug report[1] on memory leak of ParNewGC. It is solved by OpenJDK 1.6.0_20(IcedTea6 1.9.2)[2]. So the suggestion is: for who runs cassandra of Ubuntu 10.04, please upgrade OpenJDK to the latest version. [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6824570 [2] http://blog.fuseyism.com/index.php/2010/09/10/icedtea6-19-released/ best regards, hanzhu On Thu, Dec 16, 2010 at 3:10 PM, Zhu Han wrote: > The test node is behind a firewall. So I took some time to find a way to > get JMX diagnostic information from it. > > What's interesting is, both the HeapMemoryUsage and NonHeapMemoryUsage > reported by JVM is quite reasonable. So, it's a myth why the JVM process > maps such a big anonymous memory region... > > $ java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.jar - localhost:8080 > java.lang:type=Memory HeapMemoryUsage > 12/16/2010 15:07:45 +0800 org.archive.jmx.Client HeapMemoryUsage: > committed: 1065025536 > init: 1073741824 > max: 1065025536 > used: 18295328 > > $java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.jar - localhost:8080 > java.lang:type=Memory NonHeapMemoryUsage > 12/16/2010 15:01:51 +0800 org.archive.jmx.Client NonHeapMemoryUsage: > committed: 34308096 > init: 24313856 > max: 226492416 > used: 21475376 > > If anybody is interested in it, I can provide more diagnostic information > before I restart the instance. > > best regards, > hanzhu > > > > On Thu, Dec 16, 2010 at 1:00 PM, Zhu Han wrote: > >> After investigating it deeper, I suspect it's native memory leak of JVM. >> The large anonymous map on lower address space should be the native heap of >> JVM, but not java object heap. Has anybody met it before? >> >> I'll try to upgrade the JVM tonight. >> >> best regards, >> hanzhu >> >> >> >> On Thu, Dec 16, 2010 at 10:50 AM, Zhu Han wrote: >> >>> Hi, >>> >>> I have a test node with apache-cassandra-0.6.8 on ubuntu 10.4. The >>> hardware environment is an OpenVZ container. JVM settings is >>> # java -Xmx128m -version >>> java version "1.6.0_18" >>> OpenJDK Runtime Environment (IcedTea6 1.8.2) (6b18-1.8.2-4ubuntu2) >>> OpenJDK 64-Bit Server VM (build 16.0-b13, mixed mode) >>> >>> This is the memory settings: >>> >>> "/usr/bin/java -ea -Xms1G -Xmx1G ..." >>> >>> And the ondisk footprint of sstables is very small: >>> >>> "#du -sh data/ >>> "9.8M data/" >>> >>> The node was infrequently accessed in the last three weeks. After that, >>> I observe the abnormal memory utilization by top: >>> >>> PID USER PR NI *VIRT* *RES* SHR S %CPU %MEM TIME+ >>> COMMAND >>> >>> 7836 root 15 0 *3300m* *2.4g* 13m S 0 26.0 2:58.51 >>> java >>> >>> The jvm heap utilization is quite normal: >>> >>> #sudo jstat -gc -J"-Xmx128m" 7836 >>> S0C S1C S0U S1U *EC* *EU* *OC* >>> *OU* *PC PU* YGC YGCT FGC FGCT >>> GCT >>> 8512.0 8512.0 372.8 0.0 *68160.0* *5225.7* *963392.0 508200.7 >>> 30604.0 18373.4* 480 3.979 2 0.005 3.984 >>> >>> And then I try "pmap" to see the native memory mapping. *There is two >>> large anonymous mmap regions.* >>> >>> 00000000080dc000 1573568K rw--- [ anon ] >>> 00002b2afc900000 1079180K rw--- [ anon ] >>> >>> The second one should be JVM heap. What is the first one? Mmap of >>> sstable should never be anonymous mmap, but file based mmap. *Is it a >>> native memory leak? *Does cassandra allocate any DirectByteBuffer? >>> >>> best regards, >>> hanzhu >>> >> >> > --001636c598dbe1eaa9049782260c--