incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjeev Kulkarni <sanj...@locomatix.com>
Subject Re: Memory Usage During Read
Date Mon, 09 May 2011 21:44:07 GMT
Hi Adam,
We have been facing some similar issues of late. Wondering if Jonathan's
suggestions worked for you.
Thanks!

On Sat, May 7, 2011 at 6:37 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> The live:serialized size ratio depends on what your data looks like
> (small columns will be less efficient than large blobs) but using the
> rule of thumb of 10x, around 1G * (1 + memtable_flush_writers +
> memtable_flush_queue_size).
>
> So first thing I would do is drop writers and queue to 1 and 1.
>
> Then I would drop the max heap to 1G, memtable size to 8MB so the heap
> dump is easier to analyze. Then let it OOM and look at the dump with
> http://www.eclipse.org/mat/
>
> On Sat, May 7, 2011 at 3:54 PM, Serediuk, Adam
> <Adam.Serediuk@serialssolutions.com> wrote:
> > How much memory should a single hot cf with a 128mb memtable take with
> row and key caching disabled during read?
> >
> > Because I'm seeing heap go from 3.5gb skyrocketing straight to max
> (regardless of the size, 8gb and 24gb both do the same) at which time the
> jvm will do nothing but full gc and is unable to reclaim any meaningful
> amount of memory. Cassandra then becomes unusable.
> >
> > I see the same behavior with smaller memtables, eg 64mb.
> >
> > This happens well into the read operation an only on a small number of
> nodes in the cluster(1-4 out of a total of 60 nodes.)
> >
> > Sent from my iPhone
> >
> > On May 6, 2011, at 22:45, "Jonathan Ellis" <jbellis@gmail.com> wrote:
> >
> >> You don't GC storm without legitimately having a too-full heap.  It's
> >> normal to see occasional full GCs from fragmentation, but that will
> >> actually compact the heap and everything goes back to normal IF you
> >> had space actually freed up.
> >>
> >> You say you've played w/ memtable size but that would still be my bet.
> >> Most people severely underestimate how much space this takes (10x in
> >> memory over serialized size), which will bite you when you have lots
> >> of CFs defined.
> >>
> >> Otherwise, force a heap dump after a full GC and take a look to see
> >> what's referencing all the memory.
> >>
> >> On Fri, May 6, 2011 at 12:25 PM, Serediuk, Adam
> >> <Adam.Serediuk@serialssolutions.com> wrote:
> >>> We're troubleshooting a memory usage problem during batch reads. We've
> spent the last few days profiling and trying different GC settings. The
> symptoms are that after a certain amount of time during reads one or more
> nodes in the cluster will exhibit extreme memory pressure followed by a gc
> storm. We've tried every possible JVM setting and different GC methods and
> the issue persists. This is pointing towards something instantiating a lot
> of objects and keeping references so that they can't be cleaned up.
> >>>
> >>> Typically nothing is ever logged other than the GC failures however
> just now one of the nodes emitted logs we've never seen before:
> >>>
> >>>  INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085 StorageService.java
> (line 2218) Unable to reduce heap usage since there are no dirty column
> families
> >>>
> >>> We have tried increasing the heap on these nodes to large values, eg
> 24GB and still run into the same issue. We're running 8GB of heap normally
> and only one or two nodes will ever exhibit this issue, randomly. We don't
> use key/row caching and our memtable sizing is 64mb/0.3. Larger or smaller
> memtables make no difference in avoiding the issue. We're on 0.7.5, mmap,
> jna and jdk 1.6.0_24
> >>>
> >>> We've somewhat hit the wall in troubleshooting and any advice is
> greatly appreciated.
> >>>
> >>> --
> >>> Adam
> >>>
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >>
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

Mime
View raw message