cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jedd Rashbrooke <>
Subject Re: Memory leak with Sun Java 1.6 ?
Date Wed, 15 Dec 2010 16:26:24 GMT
 Hey Peter,

On 14 December 2010 20:19, Peter Schuller <> wrote:
> So, now that I get that we have two different cases ;)

 Yup.  My problem is java / environment based, occurring
 after several weeks, using 0.6.6 instances.  Our thread
 hijacking friend / learned colleague's problem is 0.7 based
 and occurs within a few minutes, and appears to be related
 to something within the JVM.  (And no .. in this case, it's
 not someone that works with me, but an understandable
 assumption given the context.)

 Flipping to standard IO mode .. might be too big a cost for
 us at the moment.  This particular system is in production,
 and it's probably too expensive to set up a parallel system,
 and run it long enough to determine if we see the same
 trends.  I'm increasingly inclined to hang out for a stable
 0.7.x release before fiddling with our production system.

> Re-reading your part of this thread, it seems you're not even having
> OOM issues in Cassandra itself, right? You're just making (1) the
> observation that memory graphs show little left, and (2) that nodetool
> etc start failing on start with OOM?

 Pretty much, yes.

 I've got two new graphs that may be of casual interest.  The first
 shows memory usage over the past two months or so, including
 the recent Cassandra restart a week ago:

 The other graph (see below) might be more insightful.

> (2) Enable GC logging with e.g. -Xloggc:/path/to/log and -XX:+PrintGC
> -XX:+PrintGCDetails -XX:+PrintGCTimestamps.

 Perhaps unrelated - though I don't suspect it was a GC problem
 (having suffered several weeks of GC pain) - Cassandra seemed
 to be happy to still do actual reads & writes while it was in this
 state.  This state meaning

> (3) Observe top to confirm JVM memory use.

 I can confirm this was showing as 7 / 4.1  for V / RES across
 all 16 boxes .. up until we did the rolling restart Cassandra.

> (4) Observe the +/- buffers in 'free' output. For example:

 Okay - here's the output of one of our 16 boxes right now:

root@prawn-07:~# free
             total       used       free     shared    buffers     cached
Mem:       7864548    7813220      51328          0      77064    2683440
-/+ buffers/cache:    5052716    2811832
Swap:            0          0          0

 Apologies for the lack of granularity in this graph, but the
 numbers at the bottom will indicate what matches what
 fairly easily.   Memory usage graph for past hour:

 This is more for your interest than anything else - I'm
 not sure there's a 'solution' for the problem as it stands,
 and I'm aware that I'm on a 2-rev old release that's about
 to be major-numbered into obscurity in the next month or
 two anyway.

> One hypothesis: Is your data set slowly increasing, and the growth in
> memory use over time is just a result of the data set  being larger,
> which would be reflected in the amount of data in page cache?

 Our data is increasing, certainly.  Much of that are overwrites, and
 we purge those after three days via GCGraceSeconds, but I don't
 have any stats on ratio or indeed raw growth rate.

> If you
> have no *other* significant activity that fills page cache (including
> log files etc), page cache would be unused otherwise and then grow in
> occupancy as your data set increases.

 Just checked, and no - /var/log is a modest 14MB, and we run nothing
 but Cassandra on these boxes.  Well, a monitoring agent, but it doesn't
 do any local logging, just some network activity.

> And to be clear: What is the actual negative observation that you have
> made - do you see high latencies, saturation on disk I/O etc or is
> this purely about seeming memory growth on graphs and observations of
> memory usage by the JVM in top?

 Ah, okay, sorry.  To reiterate:
 o  nodetool ceases to run - won't even let me drain that instance
 o  jmap won't run either
 o  a consequence of nodetool not working means that we can't monitor
     the health of our cluster, which is a scary place to be for us
 o  all 16 boxes were in the same state last Friday, as they'd all
     restarted around the same time (say 6 weeks earlier)
 o  performance on boxes approaching this state seems to be 'ok',
     but I can't give anything quantitative on this front (we don't do
     any performance monitoring calls at the moment)

 My big concern here is that because these are swapless boxes, and
 actual resident size of the java process matched (to within some tens
 of MB) the physical memory, I was very close to having all the boxes
 OOM'ing - at an OS level, I mean, not within the JVM.

 I thought (hoped?) that because we're on 7GB machines, using
 4GB Xmx, and running 0.6.6 instances for several weeks - this
 would be a common enough scenario that others might have seen
 something similar.  Alas . . .    ;)


View raw message