incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: Memory leak with Sun Java 1.6 ?
Date Fri, 10 Dec 2010 18:37:06 GMT
>  Over the past month or so, it looks like memory has slowly
>  been exhausted.  Both nodetool drain and jmap can't run, and
>  produce this error:
>     Error occurred during initialization of VM
>     Could not reserve enough space for object heap
>  We've got Xmx/Xms set to 4GB.
>  top shows free memory around 50-80MB, file cache under
>  10MB, and the java process at 12+GB virt and 7.1GB res.
>  This feels like a Java problem, not a Cassandra one, but I'm
>  open to suggestions.  To ensure I don't get bothered over
>  the weekend we're doing a rolling restart of Cassandra on
>  each of the boxes now.  The last time they were restarted
>  was just over a month ago.  Now I'm wondering whether I
>  should (until 0.7.1 is available) schedule in a slower rolling
>  restart over several days, every few weeks.

Memory-mapped files will account for both virtual and, to the extent
that they are resident in memory, to the resident size of the process.
However, your graph:

>  I've shared a Zabbix graph of system memory at:

Certainly indicates that it is not the explanation since you should be
seeing cached occupy the remainder of memory above heap size. In
addition the allocation failures from jmap indicates memory is truly

Just to confirm, what does the free +/- buffers show if you run
'free'? (I.e., middle line, under 'free' column)

A Java memory leak would likely indicate non-heap managed memory
(since I think it's unlikely that the JVM fails to limit the actual
heap size). The question is what....

To cargo cult it: Are you running a modern JVM? (Not e.g. openjdk b17
in lenny or some such.) If it is a JVM issue, ensuring you're using a
reasonably recent JVM is probably much easier than to start tracking
it down...

/ Peter Schuller

View raw message