On Sun, Dec 19, 2010 at 10:29 AM, Zhu Ha= n <schumi.han@= gmail.com> wrote:

The problem seems still like the C-heap of JVM, which leaks 70MB every day.= Here is the summary:

on 12/19: 00000000010c3000 178548K rw---=A0=A0= =A0 [ anon ]
on 12/18: 00000000010c3000 110320K rw---=A0=A0=A0 [ anon ]<= br>on 12/17: 00000000010c3000=A0 39256K rw---=A0=A0=A0 [ anon ]

This should not be the JVM object heap, because the object heap size is= fixed up per the below JVM settings. Here is the map of JVM object heap, w= hich remains constant.

00000000010c3000=A0 39256K rw---=A0=A0=A0 [ a= non ]

It should be :
00002b58433c0000 1069824K rw---=A0= =A0=A0 [ anon ]
=A0

I'll paste it to open-jdk mailist to seek for help.

Zhu,
Couple of quick questions:
=A0How many threads are in your JVM?

There = are hundreds of threads. Here is the settings of Cassandra:
1)=A0 <= ;ConcurrentReads>8</ConcurrentReads>
=A0 <ConcurrentWrites&g= t;128</ConcurrentWrites>

The thread stack size on this server is 1MB. So I observe hundreds of s= ingle mmap segment as 1MB.

=A0Can you also post the full commandline as well?
<= div>Sure. All of them are default settings.

/usr/bin/java -ea -Xms1G= -Xmx1G -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnab= led -XX:SurvivorRatio=3D8 -XX:MaxTenuringThreshold=3D1 -XX:CMSInitiatingOcc= upancyFraction=3D75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOf= MemoryError -Dcom.sun.management.jmxremote.port=3D8080 -Dcom.sun.management= .jmxremote.ssl=3Dfalse -Dcom.sun.management.jmxremote.authenticate=3Dfalse = -Dstorage-config=3Dbin/../conf -cp bin/../conf:bin/../build/classes:bin/../= lib/antlr-3.1.3.jar:bin/../lib/apache-cassandra-0.6.8.jar:bin/../lib/clhm-p= roduction.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.j= ar:bin/../lib/commons-collections-3.2.1.jar:bin/../lib/commons-lang-2.4.jar= :bin/../lib/google-collections-1.0.jar:bin/../lib/hadoop-core-0.20.1.jar:bi= n/../lib/high-scale-lib.jar:bin/../lib/ivy-2.1.0.jar:bin/../lib/jackson-cor= e-asl-1.4.0.jar:bin/../lib/jackson-mapper-asl-1.4.0.jar:bin/../lib/jline-0.= 9.94.jar:bin/../lib/jna.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libth= rift-r917130.jar:bin/../lib/log4j-1.2.14.jar:bin/../lib/slf4j-api-1.5.8.jar= :bin/../lib/slf4j-log4j12-1.5.8.jar org.apache.cassandra.thrift.CassandraDa= emon
=A0
=A0Also, output of cat /proc/meminfo

This i= s an openvz based testing environment. So /proc/meminfo is not very helpful= . Whatever, I paste it here.

MemTotal:=A0=A0=A0=A0=A0 9838380 kB=
MemFree:=A0=A0=A0=A0=A0=A0 4005900 kB
Buffers:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
Cached:=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 0 kB
SwapCached:=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 k= B
Active:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
Inactive:=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
HighTotal:=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 0 kB
HighFree:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
LowTotal:=A0= =A0=A0=A0=A0 9838380 kB
LowFree:=A0=A0=A0=A0=A0=A0 4005900 kB
SwapTotal:=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 0 kB
SwapFree:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
Dirty:=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
Writeback:=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 0 kB
AnonPages:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
Ma= pped:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
Slab:=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
PageTables:=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
NFS_Unstable:=A0=A0=A0=A0=A0= =A0=A0 0 kB
Bounce:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
Commi= tLimit:=A0=A0=A0=A0=A0=A0=A0=A0 0 kB
Committed_AS:=A0=A0=A0=A0=A0=A0=A0 = 0 kB
VmallocTotal:=A0=A0=A0=A0=A0=A0=A0 0 kB
VmallocUsed:=A0=A0=A0=A0= =A0=A0=A0=A0 0 kB
VmallocChunk:=A0=A0=A0=A0=A0=A0=A0 0 kB
HugePages_Total:=A0=A0=A0=A0 0
HugePages_Free:=A0=A0=A0=A0=A0 0
HugeP= ages_Rsvd:=A0=A0=A0=A0=A0 0
Hugepagesize:=A0=A0=A0=A0 2048 kB

thanks,
Sri

On Fri, Dec 17, 2010 at 7:15 PM, Zhu Han <schumi.han@gmail.com> wrote:

> Seems like =A0the problem there after I upgrade to "OpenJDK Runti= me
> Environment (IcedTea6 1.9.2)". So it is not related to the bug I = reported
> two days ago.
>
> Can somebody else share some info with us? What's the java environ= ment you
> used? Is it stable for long-lived cassandra instances?
>
> best regards,
> hanzhu
>
>
> On Thu, Dec 16, 2010 at 9:28 PM, Zhu Han <schumi.han@gmail.com> wrote:
>
> > I've tried it. But it does not work for me this afternoon. > >
> > Thank you!
> >
> > best regards,
> > hanzhu
> >
> >
> >
> > On Thu, Dec 16, 2010 at 8:59 PM, Matthew Conway <matt@backupify.com
> >wrote:
> >
> >> Thanks for debugging this, I'm running into the same prob= lem.
> >> BTW, if you can ssh into your nodes, you can use jconsole ove= r ssh:
> >> http://simplygenius.com/2010/08/jconsol= e-via-socks-ssh-tunnel.html
> >>
> >> Matt
> >>
> >>
> >> On Dec 16, 2010, at Thu Dec 16, 2:39 AM, Zhu Han wrote:
> >>
> >> > Sorry for spam again. :-)
> >> >
> >> > I think I find the root cause. Here is a bug report[1] o= n memory leak
> of
> >> > ParNewGC. =A0It is solved by OpenJDK 1.6.0_20(IcedTea6 1= .9.2)[2].
> >> >
> >> > So the suggestion is: for who runs cassandra =A0of Ubunt= u 10.04, please
> >> > upgrade OpenJDK to the latest version.
> >> >
> >> > [1] http://bugs.sun.com/bugdatabase/view= _bug.do?bug_id=3D6824570
> >> > [2]
> http://blog.fuseyism.com/index.php/2010/09/10/i= cedtea6-19-released/
> >> >
> >> > best regards,
> >> > hanzhu
> >> >
> >> >
> >> > On Thu, Dec 16, 2010 at 3:10 PM, Zhu Han <schumi.han@gmail.com>=
> wrote:
> >> >
> >> >> The test node is behind a firewall. So I took some t= ime to find a way
> >> to
> >> >> get JMX diagnostic information from it.
> >> >>
> >> >> What's interesting is, both the HeapMemoryUsage = and
> NonHeapMemoryUsage
> >> >> reported by JVM is quite reasonable. =A0So, it's= a myth why the JVM
> >> process
> >> >> maps such a big anonymous memory region...
> >> >>
> >> >> $ java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.j= ar -
> localhost:8080
> >> >> java.lang:type=3DMemory HeapMemoryUsage
> >> >> 12/16/2010 15:07:45 +0800 org.archive.jmx.Client Hea= pMemoryUsage:
> >> >> committed: 1065025536
> >> >> init: 1073741824
> >> >> max: 1065025536
> >> >> used: 18295328
> >> >>
> >> >> $java -Xmx128m -jar /tmp/cmdline-jmxclient-0.10.3.ja= r -
> localhost:8080
> >> >> java.lang:type=3DMemory NonHeapMemoryUsage
> >> >> 12/16/2010 15:01:51 +0800 org.archive.jmx.Client Non= HeapMemoryUsage:
> >> >> committed: 34308096
> >> >> init: 24313856
> >> >> max: 226492416
> >> >> used: 21475376
> >> >>
> >> >> If anybody is interested in it, I can provide more d= iagnostic
> >> information
> >> >> before I restart the instance.
> >> >>
> >> >> best regards,
> >> >> hanzhu
> >> >>
> >> >>
> >> >>
> >> >> On Thu, Dec 16, 2010 at 1:00 PM, Zhu Han <schumi.han@gmail.com= >
> wrote:
> >> >>
> >> >>> After investigating it deeper, =A0I suspect it&#= 39;s native memory leak of
> >> JVM.
> >> >>> The large anonymous map on lower address space s= hould be the native
> >> heap of
> >> >>> JVM, =A0but not java object heap. =A0Has anybody= met it before?
> >> >>>
> >> >>> I'll try to upgrade the JVM tonight.
> >> >>>
> >> >>> best regards,
> >> >>> hanzhu
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Thu, Dec 16, 2010 at 10:50 AM, Zhu Han <schumi.han@gmail.co= m>
> >> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> I have a test node with apache-cassandra-0.6= .8 on ubuntu 10.4. =A0The
> >> >>>> hardware environment is an OpenVZ container.= JVM settings is
> >> >>>> # java -Xmx128m -version
> >> >>>> java version "1.6.0_18"
> >> >>>> OpenJDK Runtime Environment (IcedTea6 1.8.2)= (6b18-1.8.2-4ubuntu2)
> >> >>>> OpenJDK 64-Bit Server VM (build 16.0-b13, mi= xed mode)
> >> >>>>
> >> >>>> This is the memory settings:
> >> >>>>
> >> >>>> "/usr/bin/java -ea -Xms1G -Xmx1G ...&qu= ot;
> >> >>>>
> >> >>>> And the ondisk footprint of sstables is very= small:
> >> >>>>
> >> >>>> "#du -sh data/
> >> >>>> "9.8M =A0 =A0data/"
> >> >>>>
> >> >>>> The node was infrequently accessed in the la= st =A0three weeks. =A0After
> >> that,
> >> >>>> I observe the abnormal memory utilization by= top:
> >> >>>>
> >> >>>> =A0PID USER =A0 =A0 =A0PR =A0NI =A0*VIRT* = =A0*RES* =A0SHR S %CPU %MEM =A0 =A0TIME+
> >> >>>> COMMAND
> >> >>>>
> >> >>>> 7836 root =A0 =A0 =A015 =A0 0 =A0 =A0 *3300m= * *2.4g* =A013m S =A0 =A00 26.0 =A0 2:58.51
> >> >>>> java
> >> >>>>
> >> >>>> The jvm heap utilization is quite normal: > >> >>>>
> >> >>>> #sudo jstat -gc -J"-Xmx128m" 7836<= br> > >> >>>> S0C =A0 =A0S1C =A0 =A0S0U =A0 =A0S1U =A0 =A0= =A0*EC* =A0 =A0 =A0 *EU* =A0 =A0 =A0 =A0 =A0*OC*
> >> >>>> *OU* =A0 =A0 =A0 =A0 =A0 =A0*PC =A0 =A0 =A0 = =A0 =A0 PU* =A0 =A0 =A0 =A0 =A0YGC =A0YGCT =A0FGC =A0 =A0FGCT
> >> >>>> GCT
> >> >>>> 8512.0 8512.0 372.8 =A0 0.0 =A0 *68160.0* = =A0 *5225.7* =A0 *963392.0
> >> 508200.7
> >> >>>> 30604.0 18373.4* =A0 =A0480 =A0 =A03.979 =A0= =A0 =A02 =A0 =A0 =A00.005 =A0 =A03.984
> >> >>>>
> >> >>>> And then I try "pmap" to see the n= ative memory mapping. *There is
> two
> >> >>>> large anonymous mmap regions.*
> >> >>>>
> >> >>>> 00000000080dc000 1573568K rw--- =A0 =A0[ ano= n ]
> >> >>>> 00002b2afc900000 =A01079180K rw--- =A0 =A0[ = anon ]
> >> >>>>
> >> >>>> The second one should be JVM heap. =A0What i= s the first one? =A0Mmap of
> >> >>>> sstable should never be anonymous mmap, but = file based mmap. =A0*Is
> it
> >> =A0a
> >> >>>> native memory leak? =A0*Does cassandra alloc= ate any DirectByteBuffer?
> >> >>>>
> >> >>>> best regards,
> >> >>>> hanzhu
> >> >>>>
> >> >>>
> >> >>>
> >> >>
> >>
> >>
> >
>