Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Serediuk, Adam" <Adam.Serediuk@serialssolutions.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
CC: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Sat, 7 May 2011 18:54:44 -0400
Subject: Re: Memory Usage During Read
Thread-Topic: Memory Usage During Read
Thread-Index: AcwNCcOQCGZHMDQhRgSAalr9R0dIRA==
Message-ID: <2CDF4665-8640-4050-87BD-0815E11FD23E@serialssolutions.com>
References: <564509F2-34C9-428C-901F-1A798803745A@serialssolutions.com>
 <BANLkTik+kytJDi0CCkir=D=Uio9kEmHVsg@mail.gmail.com>
In-Reply-To: <BANLkTik+kytJDi0CCkir=D=Uio9kEmHVsg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

How much memory should a single hot cf with a 128mb memtable take with row =
and key caching disabled during read?

Because I'm seeing heap go from 3.5gb skyrocketing straight to max (regardl=
ess of the size, 8gb and 24gb both do the same) at which time the jvm will =
do nothing but full gc and is unable to reclaim any meaningful amount of me=
mory. Cassandra then becomes unusable.=20

I see the same behavior with smaller memtables, eg 64mb.=20

This happens well into the read operation an only on a small number of node=
s in the cluster(1-4 out of a total of 60 nodes.)

Sent from my iPhone

On May 6, 2011, at 22:45, "Jonathan Ellis" <jbellis@gmail.com> wrote:

> You don't GC storm without legitimately having a too-full heap.  It's
> normal to see occasional full GCs from fragmentation, but that will
> actually compact the heap and everything goes back to normal IF you
> had space actually freed up.
>=20
> You say you've played w/ memtable size but that would still be my bet.
> Most people severely underestimate how much space this takes (10x in
> memory over serialized size), which will bite you when you have lots
> of CFs defined.
>=20
> Otherwise, force a heap dump after a full GC and take a look to see
> what's referencing all the memory.
>=20
> On Fri, May 6, 2011 at 12:25 PM, Serediuk, Adam
> <Adam.Serediuk@serialssolutions.com> wrote:
>> We're troubleshooting a memory usage problem during batch reads. We've s=
pent the last few days profiling and trying different GC settings. The symp=
toms are that after a certain amount of time during reads one or more nodes=
 in the cluster will exhibit extreme memory pressure followed by a gc storm=
. We've tried every possible JVM setting and different GC methods and the i=
ssue persists. This is pointing towards something instantiating a lot of ob=
jects and keeping references so that they can't be cleaned up.
>>=20
>> Typically nothing is ever logged other than the GC failures however just=
 now one of the nodes emitted logs we've never seen before:
>>=20
>>  INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085 StorageService.java (li=
ne 2218) Unable to reduce heap usage since there are no dirty column famili=
es
>>=20
>> We have tried increasing the heap on these nodes to large values, eg 24G=
B and still run into the same issue. We're running 8GB of heap normally and=
 only one or two nodes will ever exhibit this issue, randomly. We don't use=
 key/row caching and our memtable sizing is 64mb/0.3. Larger or smaller mem=
tables make no difference in avoiding the issue. We're on 0.7.5, mmap, jna =
and jdk 1.6.0_24
>>=20
>> We've somewhat hit the wall in troubleshooting and any advice is greatly=
 appreciated.
>>=20
>> --
>> Adam
>>=20
>=20
>=20
>=20
> --=20
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>=20