Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cassandra-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 209.85.219.212 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=HDgzI5+nc2vJDC5SqWVfySWRvpsEIMuMBdrRTiuLH1juf7qtKV902hMiqbLN0CcuIa
         cswChIwzIst4/awKRr9j4TJo8LQujYsvMx+HFpVhEIDmiiiBhbnZXokMlF6/Xt/QmnyY
         /VD6LxDQ6Z/sBKQk0JtVzhkX7CsXyCivpYDrM=
MIME-Version: 1.0
In-Reply-To: <20100213000434.GA16503@alumni.caltech.edu>
References: <20100211181027.GA4022@alumni.caltech.edu>
 <e06563881002111129p18358570vae02ce672b8eb18a@mail.gmail.com>
	<20100212205246.GA16171@alumni.caltech.edu>
 <e06563881002121309y40bd5403t9b44d327b28cded9@mail.gmail.com>
	<20100213000434.GA16503@alumni.caltech.edu>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Fri, 12 Feb 2010 21:24:47 -0600
Message-ID: <e06563881002121924t5090424fn405afc2ab0594ad7@mail.gmail.com>
Subject: Re: OOM on restart
To: cassandra-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

right, and remembering that MemtableSizeInMB  is just the size of the
values, you can estimate around 50% overhead on top of that.

On Fri, Feb 12, 2010 at 6:04 PM, Anthony Molinaro
<anthonym@alumni.caltech.edu> wrote:
> Okay, is it better written as
>
> NumMemtables =3D 1 + 2 * AvailableProcessors + NumDataFileDirectory
>
> Thus estimated maximum memory is
>
> MemtableMemoryUsage =3D MemtableSizeInMB * NumMemtables
>
> So in my case where I have 2 core machines with 3 datafile directories
>
> MemtableMemoryUsage =3D 512 * (1 + 2 * 2 + 3) =3D 4096MB
>
> which is basically greater than the 4G I've given to Xmx, thus the OOM?
>
> -Anthony
>
> On Fri, Feb 12, 2010 at 03:09:10PM -0600, Jonathan Ellis wrote:
>> 0.5 allows 1 + 2 * Runtime.getRuntime().availableProcessors()
>> Memtables + 1 per DataFileLocation to be waiting for flush before it
>> will block writes (or log replay) to give those time to flush out.
>> So, it sounds like you just need to lower your Memtable max
>> size/object count.
>>
>> On Fri, Feb 12, 2010 at 2:52 PM, Anthony Molinaro
>> <anthonym@alumni.caltech.edu> wrote:
>> > 0.5.0 final. =A0I was able to get things going again by upping the mem=
ory
>> > then lowering it after a successful restart, but I would like to know =
how
>> > to minimize the chances of OOM via tuning.
>> >
>> > -Anthony
>> >
>> > On Thu, Feb 11, 2010 at 01:29:16PM -0600, Jonathan Ellis wrote:
>> >> What version are you on these days? :)
>> >>
>> >> On Thu, Feb 11, 2010 at 12:11 PM, Anthony Molinaro
>> >> <anthonym@alumni.caltech.edu> wrote:
>> >> > Hi,
>> >> >
>> >> > =A0I've been having nodes failing recently with OOM exceptions (not=
 sure
>> >> > why, but we have had an increase in traffic so that could be a caus=
e).
>> >> > Most nodes have restarted fine, one node however, has been having p=
roblems
>> >> > restarting. =A0It was failing with
>> >> >
>> >> > java.lang.OutOfMemoryError: Java heap space
>> >> > =A0 =A0 =A0 =A0at java.util.Arrays.copyOfRange(Arrays.java:3209)
>> >> > =A0 =A0 =A0 =A0at java.lang.String.<init>(String.java:216)
>> >> > =A0 =A0 =A0 =A0at java.io.DataInputStream.readUTF(DataInputStream.j=
ava:644)
>> >> > =A0 =A0 =A0 =A0at java.io.DataInputStream.readUTF(DataInputStream.j=
ava:547)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilySerializer.de=
serialize(ColumnFamilySerializer.java:104)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationSerializer.def=
reezeTheMaps(RowMutation.java:308)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationSerializer.des=
erialize(RowMutation.java:318)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutationSerializer.des=
erialize(RowMutation.java:271)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.CommitLog.recover(CommitL=
og.java:338)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.RecoveryManager.doRecover=
y(RecoveryManager.java:65)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.service.CassandraDaemon.setu=
p(CassandraDaemon.java:90)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.service.CassandraDaemon.main=
(CassandraDaemon.java:166)
>> >> >
>> >> > And
>> >> >
>> >> > java.lang.OutOfMemoryError: Java heap space
>> >> > =A0 =A0 =A0 =A0at java.lang.StringCoding.encode(StringCoding.java:2=
66)
>> >> > =A0 =A0 =A0 =A0at java.lang.StringCoding.encode(StringCoding.java:2=
84)
>> >> > =A0 =A0 =A0 =A0at java.lang.String.getBytes(String.java:987)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.FBUtilities.hash(FBUti=
lities.java:178)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.dht.RandomPartitioner.getTok=
en(RandomPartitioner.java:116)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.dht.RandomPartitioner.decora=
teKey(RandomPartitioner.java:44)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.resolve(Memtable=
.java:148)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable.jav=
a:143)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.apply(C=
olumnFamilyStore.java:478)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.java:44=
5)
>> >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.CommitLog$3.run(CommitLog=
.java:365)
>> >> > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.ru=
nTask(ThreadPoolExecutor.java:886)
>> >> > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.ru=
n(ThreadPoolExecutor.java:908)
>> >> > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619)
>> >> >
>> >> > I upped the Xmx value from 4G to 6G and it seems to be doing okay, =
however
>> >> > it seems odd that it can run mostly fine with 4G, but fail to resta=
rt with
>> >> > that much memory. =A0Maybe this ticket's issue is back?
>> >> >
>> >> > https://issues.apache.org/jira/browse/CASSANDRA-609
>> >> >
>> >> > Anyway, I'm hoping thing will recover with 6G then I can restart ag=
ain with 4G and things will be good.
>> >> >
>> >> > I'd also like a better understanding of why cassandra might OOM in =
general.
>> >> > Are there settings which minimize the chances of OOM? =A0This insta=
nce has
>> >> > 2 column families and I have
>> >> >
>> >> > =A0<MemtableSizeInMB>512</MemtableSizeInMB>
>> >> > =A0<MemtableObjectCountInMillions>1.0</MemtableObjectCountInMillion=
s>
>> >> > =A0<MemtableFlushAfterMinutes>1440</MemtableFlushAfterMinutes>
>> >> >
>> >> > So if I understand these settings, memtables can at most be 512MB i=
n size
>> >> > or consist of 1 million objects before they are flushed to disk. =
=A0The maximum
>> >> > time before they will be flushed is 24 hours. =A0So does that mean =
if I fill
>> >> > up 8G or 16 memtables in less than 24 hours, I've basically used al=
l the
>> >> > memory available to me? =A0I assume there are other things using me=
mory,
>> >> > (indexes, etc), how is that limited? =A0Anyway, any information abo=
ut what
>> >> > is used where would be appreciated.
>> >> >
>> >> > Thanks,
>> >> >
>> >> > -Anthony
>> >> >
>> >> > --
>> >> > -------------------------------------------------------------------=
-----
>> >> > Anthony Molinaro =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 <anthonym@alumni.caltech.edu>
>> >> >
>> >
>> > --
>> > ----------------------------------------------------------------------=
--
>> > Anthony Molinaro =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <=
anthonym@alumni.caltech.edu>
>> >
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <ant=
honym@alumni.caltech.edu>
>