incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhu Han <schumi....@gmail.com>
Subject Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)
Date Sat, 18 Dec 2010 03:15:00 GMT
Seems like  the problem there after I upgrade to "OpenJDK Runtime
Environment (IcedTea6 1.9.2)". So it is not related to the bug I reported
two days ago.

Can somebody else share some info with us? What's the java environment you
used? Is it stable for long-lived cassandra instances?

best regards,
hanzhu


On Thu, Dec 16, 2010 at 9:28 PM, Zhu Han <schumi.han@gmail.com> wrote:

> I've tried it. But it does not work for me this afternoon.
>
> Thank you!
>
> best regards,
> hanzhu
>
>
>
> On Thu, Dec 16, 2010 at 8:59 PM, Matthew Conway <matt@backupify.com>wrote:
>
>> Thanks for debugging this, I'm running into the same problem.
>> BTW, if you can ssh into your nodes, you can use jconsole over ssh:
>> http://simplygenius.com/2010/08/jconsole-via-socks-ssh-tunnel.html
>>
>> Matt
>>
>>
>> On Dec 16, 2010, at Thu Dec 16, 2:39 AM, Zhu Han wrote:
>>
>> > Sorry for spam again. :-)
>> >
>> > I think I find the root cause. Here is a bug report[1] on memory leak of
>> > ParNewGC.  It is solved by OpenJDK 1.6.0_20(IcedTea6 1.9.2)[2].
>> >
>> > So the suggestion is: for who runs cassandra  of Ubuntu 10.04, please
>> > upgrade OpenJDK to the latest version.
>> >
>> > [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6824570
>> > [2] http://blog.fuseyism.com/index.php/2010/09/10/icedtea6-19-released/
>> >
>> > best regards,
>> > hanzhu
>> >
>> >
>> > On Thu, Dec 16, 2010 at 3:10 PM, Zhu Han <schumi.han@gmail.com> wrote:
>> >
>> >> The test node is behind a firewall. So I took some time to find a way
>> to
>> >> get JMX diagnostic information from it.
>> >>
>> >> What's interesting is, both the HeapMemoryUsage and NonHeapMemoryUsage
>> >> reported by JVM is quite reasonable.  So, it's a myth why the JVM
>> process
>> >> maps such a big anonymous memory region...
>> >>
>> >> $ java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.jar - localhost:8080
>> >> java.lang:type=Memory HeapMemoryUsage
>> >> 12/16/2010 15:07:45 +0800 org.archive.jmx.Client HeapMemoryUsage:
>> >> committed: 1065025536
>> >> init: 1073741824
>> >> max: 1065025536
>> >> used: 18295328
>> >>
>> >> $java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.jar - localhost:8080
>> >> java.lang:type=Memory NonHeapMemoryUsage
>> >> 12/16/2010 15:01:51 +0800 org.archive.jmx.Client NonHeapMemoryUsage:
>> >> committed: 34308096
>> >> init: 24313856
>> >> max: 226492416
>> >> used: 21475376
>> >>
>> >> If anybody is interested in it, I can provide more diagnostic
>> information
>> >> before I restart the instance.
>> >>
>> >> best regards,
>> >> hanzhu
>> >>
>> >>
>> >>
>> >> On Thu, Dec 16, 2010 at 1:00 PM, Zhu Han <schumi.han@gmail.com> wrote:
>> >>
>> >>> After investigating it deeper,  I suspect it's native memory leak of
>> JVM.
>> >>> The large anonymous map on lower address space should be the native
>> heap of
>> >>> JVM,  but not java object heap.  Has anybody met it before?
>> >>>
>> >>> I'll try to upgrade the JVM tonight.
>> >>>
>> >>> best regards,
>> >>> hanzhu
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Dec 16, 2010 at 10:50 AM, Zhu Han <schumi.han@gmail.com>
>> wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> I have a test node with apache-cassandra-0.6.8 on ubuntu 10.4. 
The
>> >>>> hardware environment is an OpenVZ container. JVM settings is
>> >>>> # java -Xmx128m -version
>> >>>> java version "1.6.0_18"
>> >>>> OpenJDK Runtime Environment (IcedTea6 1.8.2) (6b18-1.8.2-4ubuntu2)
>> >>>> OpenJDK 64-Bit Server VM (build 16.0-b13, mixed mode)
>> >>>>
>> >>>> This is the memory settings:
>> >>>>
>> >>>> "/usr/bin/java -ea -Xms1G -Xmx1G ..."
>> >>>>
>> >>>> And the ondisk footprint of sstables is very small:
>> >>>>
>> >>>> "#du -sh data/
>> >>>> "9.8M    data/"
>> >>>>
>> >>>> The node was infrequently accessed in the last  three weeks.  After
>> that,
>> >>>> I observe the abnormal memory utilization by top:
>> >>>>
>> >>>>  PID USER      PR  NI  *VIRT*  *RES*  SHR S %CPU %MEM    TIME+
>> >>>> COMMAND
>> >>>>
>> >>>> 7836 root      15   0     *3300m* *2.4g*  13m S    0 26.0   2:58.51
>> >>>> java
>> >>>>
>> >>>> The jvm heap utilization is quite normal:
>> >>>>
>> >>>> #sudo jstat -gc -J"-Xmx128m" 7836
>> >>>> S0C    S1C    S0U    S1U      *EC*       *EU*          *OC*
>> >>>> *OU*            *PC           PU*          YGC  YGCT  FGC    FGCT
>> >>>> GCT
>> >>>> 8512.0 8512.0 372.8   0.0   *68160.0*   *5225.7*   *963392.0
>> 508200.7
>> >>>> 30604.0 18373.4*    480    3.979      2      0.005    3.984
>> >>>>
>> >>>> And then I try "pmap" to see the native memory mapping. *There is
two
>> >>>> large anonymous mmap regions.*
>> >>>>
>> >>>> 00000000080dc000 1573568K rw---    [ anon ]
>> >>>> 00002b2afc900000  1079180K rw---    [ anon ]
>> >>>>
>> >>>> The second one should be JVM heap.  What is the first one?  Mmap
of
>> >>>> sstable should never be anonymous mmap, but file based mmap.  *Is
it
>>  a
>> >>>> native memory leak?  *Does cassandra allocate any DirectByteBuffer?
>> >>>>
>> >>>> best regards,
>> >>>> hanzhu
>> >>>>
>> >>>
>> >>>
>> >>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message