incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Coli <rc...@digg.com>
Subject Re: Out of Memory every 2 weeks
Date Mon, 14 Mar 2011 16:14:21 GMT
On Mon, Mar 14, 2011 at 8:27 AM, Jean-Yves LEBLEU <jlebleu@gmail.com> wrote:
> Sorry to create a new thread about Out of Memory problem, but I
> checked all other threads and did not find the answer.
> [...]
> The question is I don't really understand the configuration problem,
> if some body have any clue of what we could do, or what we should
> monitor to avoid the problem, as it is very difficult to reproduce the
> problem as it does not happen very often.

a) Estimate approximately how long it takes your nodes to begin to be
memory starved.
b) Wait until they are in this state, but still usable. There is
usually a "pre-fail" state where the node is just "performing very
badly" but JMX is still available.
c) "nodetool -h localhost cfstats > /tmp/my.node.with.low.free.heap.txt"
d) examine output of c) to determine which row caches and key caches
to reduce in size, in that order of priority.
e) repeat until a) no longer happens.

If you graph heap usage together with CPU usage, it will probably be
easier for you to determine when you are in state b), or are about to
be.

=Rob

Mime
View raw message