Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 361C14AEC for ; Mon, 9 May 2011 21:44:34 +0000 (UTC) Received: (qmail 54625 invoked by uid 500); 9 May 2011 21:44:32 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 54601 invoked by uid 500); 9 May 2011 21:44:32 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 54593 invoked by uid 99); 9 May 2011 21:44:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:44:32 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sanjeev@locomatix.com designates 209.85.216.179 as permitted sender) Received: from [209.85.216.179] (HELO mail-qy0-f179.google.com) (209.85.216.179) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:44:28 +0000 Received: by qyk7 with SMTP id 7so4359462qyk.10 for ; Mon, 09 May 2011 14:44:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.224.41.11 with SMTP id m11mr6579132qae.155.1304977447236; Mon, 09 May 2011 14:44:07 -0700 (PDT) Received: by 10.224.45.194 with HTTP; Mon, 9 May 2011 14:44:07 -0700 (PDT) In-Reply-To: References: <564509F2-34C9-428C-901F-1A798803745A@serialssolutions.com> <2CDF4665-8640-4050-87BD-0815E11FD23E@serialssolutions.com> Date: Mon, 9 May 2011 14:44:07 -0700 Message-ID: Subject: Re: Memory Usage During Read From: Sanjeev Kulkarni To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001485eba2d82ebc5c04a2debd7d --001485eba2d82ebc5c04a2debd7d Content-Type: text/plain; charset=ISO-8859-1 Hi Adam, We have been facing some similar issues of late. Wondering if Jonathan's suggestions worked for you. Thanks! On Sat, May 7, 2011 at 6:37 PM, Jonathan Ellis wrote: > The live:serialized size ratio depends on what your data looks like > (small columns will be less efficient than large blobs) but using the > rule of thumb of 10x, around 1G * (1 + memtable_flush_writers + > memtable_flush_queue_size). > > So first thing I would do is drop writers and queue to 1 and 1. > > Then I would drop the max heap to 1G, memtable size to 8MB so the heap > dump is easier to analyze. Then let it OOM and look at the dump with > http://www.eclipse.org/mat/ > > On Sat, May 7, 2011 at 3:54 PM, Serediuk, Adam > wrote: > > How much memory should a single hot cf with a 128mb memtable take with > row and key caching disabled during read? > > > > Because I'm seeing heap go from 3.5gb skyrocketing straight to max > (regardless of the size, 8gb and 24gb both do the same) at which time the > jvm will do nothing but full gc and is unable to reclaim any meaningful > amount of memory. Cassandra then becomes unusable. > > > > I see the same behavior with smaller memtables, eg 64mb. > > > > This happens well into the read operation an only on a small number of > nodes in the cluster(1-4 out of a total of 60 nodes.) > > > > Sent from my iPhone > > > > On May 6, 2011, at 22:45, "Jonathan Ellis" wrote: > > > >> You don't GC storm without legitimately having a too-full heap. It's > >> normal to see occasional full GCs from fragmentation, but that will > >> actually compact the heap and everything goes back to normal IF you > >> had space actually freed up. > >> > >> You say you've played w/ memtable size but that would still be my bet. > >> Most people severely underestimate how much space this takes (10x in > >> memory over serialized size), which will bite you when you have lots > >> of CFs defined. > >> > >> Otherwise, force a heap dump after a full GC and take a look to see > >> what's referencing all the memory. > >> > >> On Fri, May 6, 2011 at 12:25 PM, Serediuk, Adam > >> wrote: > >>> We're troubleshooting a memory usage problem during batch reads. We've > spent the last few days profiling and trying different GC settings. The > symptoms are that after a certain amount of time during reads one or more > nodes in the cluster will exhibit extreme memory pressure followed by a gc > storm. We've tried every possible JVM setting and different GC methods and > the issue persists. This is pointing towards something instantiating a lot > of objects and keeping references so that they can't be cleaned up. > >>> > >>> Typically nothing is ever logged other than the GC failures however > just now one of the nodes emitted logs we've never seen before: > >>> > >>> INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085 StorageService.java > (line 2218) Unable to reduce heap usage since there are no dirty column > families > >>> > >>> We have tried increasing the heap on these nodes to large values, eg > 24GB and still run into the same issue. We're running 8GB of heap normally > and only one or two nodes will ever exhibit this issue, randomly. We don't > use key/row caching and our memtable sizing is 64mb/0.3. Larger or smaller > memtables make no difference in avoiding the issue. We're on 0.7.5, mmap, > jna and jdk 1.6.0_24 > >>> > >>> We've somewhat hit the wall in troubleshooting and any advice is > greatly appreciated. > >>> > >>> -- > >>> Adam > >>> > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of DataStax, the source for professional Cassandra support > >> http://www.datastax.com > >> > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > --001485eba2d82ebc5c04a2debd7d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Adam,
We have been facing some similar issues of late. Wondering if = Jonathan's suggestions worked for you.
Thanks!

On Sat, May 7, 2011 at 6:37 PM, Jonathan Ellis <jbellis@gmail.com>= ; wrote:
The live:serialized size ratio depends on w= hat your data looks like
(small columns will be less efficient than large blobs) but using the
rule of thumb of 10x, around 1G * (1 + memtable_flush_writers +
memtable_flush_queue_size).

So first thing I would do is drop writers and queue to 1 and 1.

Then I would drop the max heap to 1G, memtable size to 8MB so the heap
dump is easier to analyze. Then let it OOM and look at the dump with
http://www.eclips= e.org/mat/

On Sat, May 7, 2011 at 3:54 PM, Serediuk, Adam
<Adam.Serediuk@serialssolutions.com> wrote:
> How much memory should a single hot cf with a 128mb memtable take with= row and key caching disabled during read?
>
> Because I'm seeing heap go from 3.5gb skyrocketing straight to max= (regardless of the size, 8gb and 24gb both do the same) at which time the = jvm will do nothing but full gc and is unable to reclaim any meaningful amo= unt of memory. Cassandra then becomes unusable.
>
> I see the same behavior with smaller memtables, eg 64mb.
>
> This happens well into the read operation an only on a small number of= nodes in the cluster(1-4 out of a total of 60 nodes.)
>
> Sent from my iPhone
>
> On May 6, 2011, at 22:45, "Jonathan Ellis" <jbellis@gmail.com> wrote:
>
>> You don't GC storm without legitimately having a too-full heap= . =A0It's
>> normal to see occasional full GCs from fragmentation, but that wil= l
>> actually compact the heap and everything goes back to normal IF yo= u
>> had space actually freed up.
>>
>> You say you've played w/ memtable size but that would still be= my bet.
>> Most people severely underestimate how much space this takes (10x = in
>> memory over serialized size), which will bite you when you have lo= ts
>> of CFs defined.
>>
>> Otherwise, force a heap dump after a full GC and take a look to se= e
>> what's referencing all the memory.
>>
>> On Fri, May 6, 2011 at 12:25 PM, Serediuk, Adam
>> <Adam.Ser= ediuk@serialssolutions.com> wrote:
>>> We're troubleshooting a memory usage problem during batch = reads. We've spent the last few days profiling and trying different GC = settings. The symptoms are that after a certain amount of time during reads= one or more nodes in the cluster will exhibit extreme memory pressure foll= owed by a gc storm. We've tried every possible JVM setting and differen= t GC methods and the issue persists. This is pointing towards something ins= tantiating a lot of objects and keeping references so that they can't b= e cleaned up.
>>>
>>> Typically nothing is ever logged other than the GC failures ho= wever just now one of the nodes emitted logs we've never seen before: >>>
>>> =A0INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085 StorageServ= ice.java (line 2218) Unable to reduce heap usage since there are no dirty c= olumn families
>>>
>>> We have tried increasing the heap on these nodes to large valu= es, eg 24GB and still run into the same issue. We're running 8GB of hea= p normally and only one or two nodes will ever exhibit this issue, randomly= . We don't use key/row caching and our memtable sizing is 64mb/0.3. Lar= ger or smaller memtables make no difference in avoiding the issue. We'r= e on 0.7.5, mmap, jna and jdk 1.6.0_24
>>>
>>> We've somewhat hit the wall in troubleshooting and any adv= ice is greatly appreciated.
>>>
>>> --
>>> Adam
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra supp= ort
>> http://www.d= atastax.com
>>
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.c= om

--001485eba2d82ebc5c04a2debd7d--