cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Memory Usage During Read
Date Sat, 07 May 2011 05:44:44 GMT
You don't GC storm without legitimately having a too-full heap.  It's
normal to see occasional full GCs from fragmentation, but that will
actually compact the heap and everything goes back to normal IF you
had space actually freed up.

You say you've played w/ memtable size but that would still be my bet.
Most people severely underestimate how much space this takes (10x in
memory over serialized size), which will bite you when you have lots
of CFs defined.

Otherwise, force a heap dump after a full GC and take a look to see
what's referencing all the memory.

On Fri, May 6, 2011 at 12:25 PM, Serediuk, Adam
<Adam.Serediuk@serialssolutions.com> wrote:
> We're troubleshooting a memory usage problem during batch reads. We've spent the last
few days profiling and trying different GC settings. The symptoms are that after a certain
amount of time during reads one or more nodes in the cluster will exhibit extreme memory pressure
followed by a gc storm. We've tried every possible JVM setting and different GC methods and
the issue persists. This is pointing towards something instantiating a lot of objects and
keeping references so that they can't be cleaned up.
>
> Typically nothing is ever logged other than the GC failures however just now one of the
nodes emitted logs we've never seen before:
>
>  INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085 StorageService.java (line 2218) Unable
to reduce heap usage since there are no dirty column families
>
> We have tried increasing the heap on these nodes to large values, eg 24GB and still run
into the same issue. We're running 8GB of heap normally and only one or two nodes will ever
exhibit this issue, randomly. We don't use key/row caching and our memtable sizing is 64mb/0.3.
Larger or smaller memtables make no difference in avoiding the issue. We're on 0.7.5, mmap,
jna and jdk 1.6.0_24
>
> We've somewhat hit the wall in troubleshooting and any advice is greatly appreciated.
>
> --
> Adam
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message