Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 209.85.219.212 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=JKF13Gwt3YxLq+mE2DTD8C5n0m2NV4AVFYdZoJah3ObcqnR7pJksn6hCF6t/MLPugh
         x4AhQ70kbtxKtzDnl+iisxmiaPGo9Tt/l3gfXWB+o6Zlt1op5fly3kMK0jNh51rnekqZ
         7s92EzEva/O0HBYFRE1I4HGlb+7nLWkSA6cno=
MIME-Version: 1.0
In-Reply-To: <20100212205246.GA16171@alumni.caltech.edu>
References: <20100211181027.GA4022@alumni.caltech.edu>
 <e06563881002111129p18358570vae02ce672b8eb18a@mail.gmail.com>
	<20100212205246.GA16171@alumni.caltech.edu>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Fri, 12 Feb 2010 15:09:10 -0600
Message-ID: <e06563881002121309y40bd5403t9b44d327b28cded9@mail.gmail.com>
Subject: Re: OOM on restart
To: cassandra-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

0.5 allows 1 + 2 * Runtime.getRuntime().availableProcessors()
Memtables + 1 per DataFileLocation to be waiting for flush before it
will block writes (or log replay) to give those time to flush out.
So, it sounds like you just need to lower your Memtable max
size/object count.

On Fri, Feb 12, 2010 at 2:52 PM, Anthony Molinaro
<anthonym@alumni.caltech.edu> wrote:
> 0.5.0 final. =A0I was able to get things going again by upping the memory
> then lowering it after a successful restart, but I would like to know how
> to minimize the chances of OOM via tuning.
>
> -Anthony
>
> On Thu, Feb 11, 2010 at 01:29:16PM -0600, Jonathan Ellis wrote:
>> What version are you on these days? :)
>>
>> On Thu, Feb 11, 2010 at 12:11 PM, Anthony Molinaro
>> <anthonym@alumni.caltech.edu> wrote:
>> > Hi,
>> >
>> > =A0I've been having nodes failing recently with OOM exceptions (not su=
re
>> > why, but we have had an increase in traffic so that could be a cause).
>> > Most nodes have restarted fine, one node however, has been having prob=
lems
>> > restarting. =A0It was failing with
>> >
>> > java.lang.OutOfMemoryError: Java heap space
>> > =A0 =A0 =A0 =A0at java.util.Arrays.copyOfRange(Arrays.java:3209)
>> > =A0 =A0 =A0 =A0at java.lang.String.<init>(String.java:216)
>> > =A0 =A0 =A0 =A0at java.io.DataInputStream.readUTF(DataInputStream.java=
:644)
>> > =A0 =A0 =A0 =A0at java.io.DataInputStream.readUTF(DataInputStream.java=
:547)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilySerializer.deser=
ialize(ColumnFamilySerializer.java:104)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationSerializer.defree=
zeTheMaps(RowMutation.java:308)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationSerializer.deseri=
alize(RowMutation.java:318)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationSerializer.deseri=
alize(RowMutation.java:271)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.CommitLog.recover(CommitLog.=
java:338)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.RecoveryManager.doRecovery(R=
ecoveryManager.java:65)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.service.CassandraDaemon.setup(C=
assandraDaemon.java:90)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.service.CassandraDaemon.main(Ca=
ssandraDaemon.java:166)
>> >
>> > And
>> >
>> > java.lang.OutOfMemoryError: Java heap space
>> > =A0 =A0 =A0 =A0at java.lang.StringCoding.encode(StringCoding.java:266)
>> > =A0 =A0 =A0 =A0at java.lang.StringCoding.encode(StringCoding.java:284)
>> > =A0 =A0 =A0 =A0at java.lang.String.getBytes(String.java:987)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.FBUtilities.hash(FBUtilit=
ies.java:178)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.dht.RandomPartitioner.getToken(=
RandomPartitioner.java:116)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.dht.RandomPartitioner.decorateK=
ey(RandomPartitioner.java:44)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.resolve(Memtable.ja=
va:148)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable.java:1=
43)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.apply(Colu=
mnFamilyStore.java:478)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.java:445)
>> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.CommitLog$3.run(CommitLog.ja=
va:365)
>> > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTa=
sk(ThreadPoolExecutor.java:886)
>> > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(T=
hreadPoolExecutor.java:908)
>> > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619)
>> >
>> > I upped the Xmx value from 4G to 6G and it seems to be doing okay, how=
ever
>> > it seems odd that it can run mostly fine with 4G, but fail to restart =
with
>> > that much memory. =A0Maybe this ticket's issue is back?
>> >
>> > https://issues.apache.org/jira/browse/CASSANDRA-609
>> >
>> > Anyway, I'm hoping thing will recover with 6G then I can restart again=
 with 4G and things will be good.
>> >
>> > I'd also like a better understanding of why cassandra might OOM in gen=
eral.
>> > Are there settings which minimize the chances of OOM? =A0This instance=
 has
>> > 2 column families and I have
>> >
>> > =A0<MemtableSizeInMB>512</MemtableSizeInMB>
>> > =A0<MemtableObjectCountInMillions>1.0</MemtableObjectCountInMillions>
>> > =A0<MemtableFlushAfterMinutes>1440</MemtableFlushAfterMinutes>
>> >
>> > So if I understand these settings, memtables can at most be 512MB in s=
ize
>> > or consist of 1 million objects before they are flushed to disk. =A0Th=
e maximum
>> > time before they will be flushed is 24 hours. =A0So does that mean if =
I fill
>> > up 8G or 16 memtables in less than 24 hours, I've basically used all t=
he
>> > memory available to me? =A0I assume there are other things using memor=
y,
>> > (indexes, etc), how is that limited? =A0Anyway, any information about =
what
>> > is used where would be appreciated.
>> >
>> > Thanks,
>> >
>> > -Anthony
>> >
>> > --
>> > ----------------------------------------------------------------------=
--
>> > Anthony Molinaro =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <=
anthonym@alumni.caltech.edu>
>> >
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <ant=
honym@alumni.caltech.edu>
>