incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SriSatish Ambati <srisatish.amb...@gmail.com>
Subject Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)
Date Sat, 18 Dec 2010 22:06:35 GMT
Zhu,
Couple of quick questions:
  How many threads are in your JVM?
  Can you also post the full commandline as well?
  Also, output of cat /proc/meminfo

thanks,
Sri

On Fri, Dec 17, 2010 at 7:15 PM, Zhu Han <schumi.han@gmail.com> wrote:

> Seems like  the problem there after I upgrade to "OpenJDK Runtime
> Environment (IcedTea6 1.9.2)". So it is not related to the bug I reported
> two days ago.
>
> Can somebody else share some info with us? What's the java environment you
> used? Is it stable for long-lived cassandra instances?
>
> best regards,
> hanzhu
>
>
> On Thu, Dec 16, 2010 at 9:28 PM, Zhu Han <schumi.han@gmail.com> wrote:
>
> > I've tried it. But it does not work for me this afternoon.
> >
> > Thank you!
> >
> > best regards,
> > hanzhu
> >
> >
> >
> > On Thu, Dec 16, 2010 at 8:59 PM, Matthew Conway <matt@backupify.com
> >wrote:
> >
> >> Thanks for debugging this, I'm running into the same problem.
> >> BTW, if you can ssh into your nodes, you can use jconsole over ssh:
> >> http://simplygenius.com/2010/08/jconsole-via-socks-ssh-tunnel.html
> >>
> >> Matt
> >>
> >>
> >> On Dec 16, 2010, at Thu Dec 16, 2:39 AM, Zhu Han wrote:
> >>
> >> > Sorry for spam again. :-)
> >> >
> >> > I think I find the root cause. Here is a bug report[1] on memory leak
> of
> >> > ParNewGC.  It is solved by OpenJDK 1.6.0_20(IcedTea6 1.9.2)[2].
> >> >
> >> > So the suggestion is: for who runs cassandra  of Ubuntu 10.04, please
> >> > upgrade OpenJDK to the latest version.
> >> >
> >> > [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6824570
> >> > [2]
> http://blog.fuseyism.com/index.php/2010/09/10/icedtea6-19-released/
> >> >
> >> > best regards,
> >> > hanzhu
> >> >
> >> >
> >> > On Thu, Dec 16, 2010 at 3:10 PM, Zhu Han <schumi.han@gmail.com>
> wrote:
> >> >
> >> >> The test node is behind a firewall. So I took some time to find a way
> >> to
> >> >> get JMX diagnostic information from it.
> >> >>
> >> >> What's interesting is, both the HeapMemoryUsage and
> NonHeapMemoryUsage
> >> >> reported by JVM is quite reasonable.  So, it's a myth why the JVM
> >> process
> >> >> maps such a big anonymous memory region...
> >> >>
> >> >> $ java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.jar -
> localhost:8080
> >> >> java.lang:type=Memory HeapMemoryUsage
> >> >> 12/16/2010 15:07:45 +0800 org.archive.jmx.Client HeapMemoryUsage:
> >> >> committed: 1065025536
> >> >> init: 1073741824
> >> >> max: 1065025536
> >> >> used: 18295328
> >> >>
> >> >> $java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.jar -
> localhost:8080
> >> >> java.lang:type=Memory NonHeapMemoryUsage
> >> >> 12/16/2010 15:01:51 +0800 org.archive.jmx.Client NonHeapMemoryUsage:
> >> >> committed: 34308096
> >> >> init: 24313856
> >> >> max: 226492416
> >> >> used: 21475376
> >> >>
> >> >> If anybody is interested in it, I can provide more diagnostic
> >> information
> >> >> before I restart the instance.
> >> >>
> >> >> best regards,
> >> >> hanzhu
> >> >>
> >> >>
> >> >>
> >> >> On Thu, Dec 16, 2010 at 1:00 PM, Zhu Han <schumi.han@gmail.com>
> wrote:
> >> >>
> >> >>> After investigating it deeper,  I suspect it's native memory leak
of
> >> JVM.
> >> >>> The large anonymous map on lower address space should be the native
> >> heap of
> >> >>> JVM,  but not java object heap.  Has anybody met it before?
> >> >>>
> >> >>> I'll try to upgrade the JVM tonight.
> >> >>>
> >> >>> best regards,
> >> >>> hanzhu
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Thu, Dec 16, 2010 at 10:50 AM, Zhu Han <schumi.han@gmail.com>
> >> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> I have a test node with apache-cassandra-0.6.8 on ubuntu 10.4.
 The
> >> >>>> hardware environment is an OpenVZ container. JVM settings is
> >> >>>> # java -Xmx128m -version
> >> >>>> java version "1.6.0_18"
> >> >>>> OpenJDK Runtime Environment (IcedTea6 1.8.2) (6b18-1.8.2-4ubuntu2)
> >> >>>> OpenJDK 64-Bit Server VM (build 16.0-b13, mixed mode)
> >> >>>>
> >> >>>> This is the memory settings:
> >> >>>>
> >> >>>> "/usr/bin/java -ea -Xms1G -Xmx1G ..."
> >> >>>>
> >> >>>> And the ondisk footprint of sstables is very small:
> >> >>>>
> >> >>>> "#du -sh data/
> >> >>>> "9.8M    data/"
> >> >>>>
> >> >>>> The node was infrequently accessed in the last  three weeks.
 After
> >> that,
> >> >>>> I observe the abnormal memory utilization by top:
> >> >>>>
> >> >>>>  PID USER      PR  NI  *VIRT*  *RES*  SHR S %CPU %MEM    TIME+
> >> >>>> COMMAND
> >> >>>>
> >> >>>> 7836 root      15   0     *3300m* *2.4g*  13m S    0 26.0 
 2:58.51
> >> >>>> java
> >> >>>>
> >> >>>> The jvm heap utilization is quite normal:
> >> >>>>
> >> >>>> #sudo jstat -gc -J"-Xmx128m" 7836
> >> >>>> S0C    S1C    S0U    S1U      *EC*       *EU*          *OC*
> >> >>>> *OU*            *PC           PU*          YGC  YGCT  FGC 
  FGCT
> >> >>>> GCT
> >> >>>> 8512.0 8512.0 372.8   0.0   *68160.0*   *5225.7*   *963392.0
> >> 508200.7
> >> >>>> 30604.0 18373.4*    480    3.979      2      0.005    3.984
> >> >>>>
> >> >>>> And then I try "pmap" to see the native memory mapping. *There
is
> two
> >> >>>> large anonymous mmap regions.*
> >> >>>>
> >> >>>> 00000000080dc000 1573568K rw---    [ anon ]
> >> >>>> 00002b2afc900000  1079180K rw---    [ anon ]
> >> >>>>
> >> >>>> The second one should be JVM heap.  What is the first one?
 Mmap of
> >> >>>> sstable should never be anonymous mmap, but file based mmap.
 *Is
> it
> >>  a
> >> >>>> native memory leak?  *Does cassandra allocate any DirectByteBuffer?
> >> >>>>
> >> >>>> best regards,
> >> >>>> hanzhu
> >> >>>>
> >> >>>
> >> >>>
> >> >>
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message