cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: Re: Cassandra 0.7 beta 3 outOfMemory (OOM)
Date Mon, 06 Dec 2010 08:22:25 GMT
> In our insert-tests the average heap usage is slowly growing up to the 3 GB
> border (jconsole monitor over 50 min http://oi51.tinypic.com/k12gzd.jpg) and
> the CompactionManger queue is also constantly growing up to about 50 jobs
> pending.

Since you're obviously bottlenecking on compaction; are you building
up lots of memtables flushes that don't complete? (I don't remember
the name of the stage off hand, but it should be visible in cfstats).

Also, if you simply stop writing suddenly and wait for the nodes to
finish doing background activies, does memory usage go down again?

(You may want to force a full GC before/after in order to do a proper
test that is not affected by GC scheduling.)

I don't remember the switches to JRockit, but you can definitely
enable GC logging there which should tell you in more detail what's
happening. IIRC, though possibly not for all GC modes, you should see
periodic completions of concurrent GC:s that should collect all
garbage that existed at the beginning of the GC cycle. Assuming you're
not under so much load that this takes a very long time, that should
give you a pretty good idea of the actual live set (which is probably
going to be the low dips in your graphs, but it doesn't hurt
confirming).

-- 
/ Peter Schuller

Mime
View raw message