incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhu Han <schumi....@gmail.com>
Subject Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)
Date Thu, 16 Dec 2010 13:28:03 GMT
I've tried it. But it does not work for me this afternoon.

Thank you!

best regards,
hanzhu


On Thu, Dec 16, 2010 at 8:59 PM, Matthew Conway <matt@backupify.com> wrote:

> Thanks for debugging this, I'm running into the same problem.
> BTW, if you can ssh into your nodes, you can use jconsole over ssh:
> http://simplygenius.com/2010/08/jconsole-via-socks-ssh-tunnel.html
>
> Matt
>
>
> On Dec 16, 2010, at Thu Dec 16, 2:39 AM, Zhu Han wrote:
>
> > Sorry for spam again. :-)
> >
> > I think I find the root cause. Here is a bug report[1] on memory leak of
> > ParNewGC.  It is solved by OpenJDK 1.6.0_20(IcedTea6 1.9.2)[2].
> >
> > So the suggestion is: for who runs cassandra  of Ubuntu 10.04, please
> > upgrade OpenJDK to the latest version.
> >
> > [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6824570
> > [2] http://blog.fuseyism.com/index.php/2010/09/10/icedtea6-19-released/
> >
> > best regards,
> > hanzhu
> >
> >
> > On Thu, Dec 16, 2010 at 3:10 PM, Zhu Han <schumi.han@gmail.com> wrote:
> >
> >> The test node is behind a firewall. So I took some time to find a way to
> >> get JMX diagnostic information from it.
> >>
> >> What's interesting is, both the HeapMemoryUsage and NonHeapMemoryUsage
> >> reported by JVM is quite reasonable.  So, it's a myth why the JVM
> process
> >> maps such a big anonymous memory region...
> >>
> >> $ java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.jar - localhost:8080
> >> java.lang:type=Memory HeapMemoryUsage
> >> 12/16/2010 15:07:45 +0800 org.archive.jmx.Client HeapMemoryUsage:
> >> committed: 1065025536
> >> init: 1073741824
> >> max: 1065025536
> >> used: 18295328
> >>
> >> $java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.jar - localhost:8080
> >> java.lang:type=Memory NonHeapMemoryUsage
> >> 12/16/2010 15:01:51 +0800 org.archive.jmx.Client NonHeapMemoryUsage:
> >> committed: 34308096
> >> init: 24313856
> >> max: 226492416
> >> used: 21475376
> >>
> >> If anybody is interested in it, I can provide more diagnostic
> information
> >> before I restart the instance.
> >>
> >> best regards,
> >> hanzhu
> >>
> >>
> >>
> >> On Thu, Dec 16, 2010 at 1:00 PM, Zhu Han <schumi.han@gmail.com> wrote:
> >>
> >>> After investigating it deeper,  I suspect it's native memory leak of
> JVM.
> >>> The large anonymous map on lower address space should be the native
> heap of
> >>> JVM,  but not java object heap.  Has anybody met it before?
> >>>
> >>> I'll try to upgrade the JVM tonight.
> >>>
> >>> best regards,
> >>> hanzhu
> >>>
> >>>
> >>>
> >>> On Thu, Dec 16, 2010 at 10:50 AM, Zhu Han <schumi.han@gmail.com>
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I have a test node with apache-cassandra-0.6.8 on ubuntu 10.4.  The
> >>>> hardware environment is an OpenVZ container. JVM settings is
> >>>> # java -Xmx128m -version
> >>>> java version "1.6.0_18"
> >>>> OpenJDK Runtime Environment (IcedTea6 1.8.2) (6b18-1.8.2-4ubuntu2)
> >>>> OpenJDK 64-Bit Server VM (build 16.0-b13, mixed mode)
> >>>>
> >>>> This is the memory settings:
> >>>>
> >>>> "/usr/bin/java -ea -Xms1G -Xmx1G ..."
> >>>>
> >>>> And the ondisk footprint of sstables is very small:
> >>>>
> >>>> "#du -sh data/
> >>>> "9.8M    data/"
> >>>>
> >>>> The node was infrequently accessed in the last  three weeks.  After
> that,
> >>>> I observe the abnormal memory utilization by top:
> >>>>
> >>>>  PID USER      PR  NI  *VIRT*  *RES*  SHR S %CPU %MEM    TIME+
> >>>> COMMAND
> >>>>
> >>>> 7836 root      15   0     *3300m* *2.4g*  13m S    0 26.0   2:58.51
> >>>> java
> >>>>
> >>>> The jvm heap utilization is quite normal:
> >>>>
> >>>> #sudo jstat -gc -J"-Xmx128m" 7836
> >>>> S0C    S1C    S0U    S1U      *EC*       *EU*          *OC*
> >>>> *OU*            *PC           PU*          YGC  YGCT  FGC    FGCT
> >>>> GCT
> >>>> 8512.0 8512.0 372.8   0.0   *68160.0*   *5225.7*   *963392.0
> 508200.7
> >>>> 30604.0 18373.4*    480    3.979      2      0.005    3.984
> >>>>
> >>>> And then I try "pmap" to see the native memory mapping. *There is two
> >>>> large anonymous mmap regions.*
> >>>>
> >>>> 00000000080dc000 1573568K rw---    [ anon ]
> >>>> 00002b2afc900000  1079180K rw---    [ anon ]
> >>>>
> >>>> The second one should be JVM heap.  What is the first one?  Mmap of
> >>>> sstable should never be anonymous mmap, but file based mmap.  *Is it
>  a
> >>>> native memory leak?  *Does cassandra allocate any DirectByteBuffer?
> >>>>
> >>>> best regards,
> >>>> hanzhu
> >>>>
> >>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message