cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sris <srisatish.amb...@gmail.com>
Subject Re: Memory Usage During Read
Date Sat, 14 May 2011 10:20:46 GMT
>>>> Typically nothing is ever logged other than the GC failures

In addition to the heapdumps,
be useful to see some GC logs
  (turn on GC logs via  cassandra.in.sh
Or add
-Xloggc:/var/log/cassandra/gc.log
-XX:+PrintGCDetails
  )

thanks, Sri
On May 7, 2011, at 6:37 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> The live:serialized size ratio depends on what your data looks like
> (small columns will be less efficient than large blobs) but using the
> rule of thumb of 10x, around 1G * (1 + memtable_flush_writers +
> memtable_flush_queue_size).
>
> So first thing I would do is drop writers and queue to 1 and 1.
>
> Then I would drop the max heap to 1G, memtable size to 8MB so the heap
> dump is easier to analyze. Then let it OOM and look at the dump with
> http://www.eclipse.org/mat/
>
> On Sat, May 7, 2011 at 3:54 PM, Serediuk, Adam
> <Adam.Serediuk@serialssolutions.com> wrote:
>> How much memory should a single hot cf with a 128mb memtable take  
>> with row and key caching disabled during read?
>>
>> Because I'm seeing heap go from 3.5gb skyrocketing straight to max  
>> (regardless of the size, 8gb and 24gb both do the same) at which  
>> time the jvm will do nothing but full gc and is unable to reclaim  
>> any meaningful amount of memory. Cassandra then becomes unusable.
>>
>> I see the same behavior with smaller memtables, eg 64mb.
>>
>> This happens well into the read operation an only on a small number  
>> of nodes in the cluster(1-4 out of a total of 60 nodes.)
>>
>> Sent from my iPhone
>>
>> On May 6, 2011, at 22:45, "Jonathan Ellis" <jbellis@gmail.com> wrote:
>>
>>> You don't GC storm without legitimately having a too-full heap.   
>>> It's
>>> normal to see occasional full GCs from fragmentation, but that will
>>> actually compact the heap and everything goes back to normal IF you
>>> had space actually freed up.
>>>
>>> You say you've played w/ memtable size but that would still be my  
>>> bet.
>>> Most people severely underestimate how much space this takes (10x in
>>> memory over serialized size), which will bite you when you have lots
>>> of CFs defined.
>>>
>>> Otherwise, force a heap dump after a full GC and take a look to see
>>> what's referencing all the memory.
>>>
>>> On Fri, May 6, 2011 at 12:25 PM, Serediuk, Adam
>>> <Adam.Serediuk@serialssolutions.com> wrote:
>>>> We're troubleshooting a memory usage problem during batch reads.  
>>>> We've spent the last few days profiling and trying different GC  
>>>> settings. The symptoms are that after a certain amount of time  
>>>> during reads one or more nodes in the cluster will exhibit  
>>>> extreme memory pressure followed by a gc storm. We've tried every  
>>>> possible JVM setting and different GC methods and the issue  
>>>> persists. This is pointing towards something instantiating a lot  
>>>> of objects and keeping references so that they can't be cleaned up.
>>>>
>>>> Typically nothing is ever logged other than the GC failures  
>>>> however just now one of the nodes emitted logs we've never seen  
>>>> before:
>>>>
>>>>  INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085  
>>>> StorageService.java (line 2218) Unable to reduce heap usage since  
>>>> there are no dirty column families
>>>>
>>>> We have tried increasing the heap on these nodes to large values,  
>>>> eg 24GB and still run into the same issue. We're running 8GB of  
>>>> heap normally and only one or two nodes will ever exhibit this  
>>>> issue, randomly. We don't use key/row caching and our memtable  
>>>> sizing is 64mb/0.3. Larger or smaller memtables make no  
>>>> difference in avoiding the issue. We're on 0.7.5, mmap, jna and  
>>>> jdk 1.6.0_24
>>>>
>>>> We've somewhat hit the wall in troubleshooting and any advice is  
>>>> greatly appreciated.
>>>>
>>>> --
>>>> Adam
>>>>
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of DataStax, the source for professional Cassandra  
>>> support
>>> http://www.datastax.com
>>>
>>
>>
>
>
>
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com

Mime
View raw message