cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: OOM on restart
Date Sat, 13 Feb 2010 03:24:47 GMT
right, and remembering that MemtableSizeInMB  is just the size of the
values, you can estimate around 50% overhead on top of that.

On Fri, Feb 12, 2010 at 6:04 PM, Anthony Molinaro
<anthonym@alumni.caltech.edu> wrote:
> Okay, is it better written as
>
> NumMemtables = 1 + 2 * AvailableProcessors + NumDataFileDirectory
>
> Thus estimated maximum memory is
>
> MemtableMemoryUsage = MemtableSizeInMB * NumMemtables
>
> So in my case where I have 2 core machines with 3 datafile directories
>
> MemtableMemoryUsage = 512 * (1 + 2 * 2 + 3) = 4096MB
>
> which is basically greater than the 4G I've given to Xmx, thus the OOM?
>
> -Anthony
>
> On Fri, Feb 12, 2010 at 03:09:10PM -0600, Jonathan Ellis wrote:
>> 0.5 allows 1 + 2 * Runtime.getRuntime().availableProcessors()
>> Memtables + 1 per DataFileLocation to be waiting for flush before it
>> will block writes (or log replay) to give those time to flush out.
>> So, it sounds like you just need to lower your Memtable max
>> size/object count.
>>
>> On Fri, Feb 12, 2010 at 2:52 PM, Anthony Molinaro
>> <anthonym@alumni.caltech.edu> wrote:
>> > 0.5.0 final.  I was able to get things going again by upping the memory
>> > then lowering it after a successful restart, but I would like to know how
>> > to minimize the chances of OOM via tuning.
>> >
>> > -Anthony
>> >
>> > On Thu, Feb 11, 2010 at 01:29:16PM -0600, Jonathan Ellis wrote:
>> >> What version are you on these days? :)
>> >>
>> >> On Thu, Feb 11, 2010 at 12:11 PM, Anthony Molinaro
>> >> <anthonym@alumni.caltech.edu> wrote:
>> >> > Hi,
>> >> >
>> >> >  I've been having nodes failing recently with OOM exceptions (not
sure
>> >> > why, but we have had an increase in traffic so that could be a cause).
>> >> > Most nodes have restarted fine, one node however, has been having problems
>> >> > restarting.  It was failing with
>> >> >
>> >> > java.lang.OutOfMemoryError: Java heap space
>> >> >        at java.util.Arrays.copyOfRange(Arrays.java:3209)
>> >> >        at java.lang.String.<init>(String.java:216)
>> >> >        at java.io.DataInputStream.readUTF(DataInputStream.java:644)
>> >> >        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>> >> >        at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:104)
>> >> >        at org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:308)
>> >> >        at org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:318)
>> >> >        at org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:271)
>> >> >        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:338)
>> >> >        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65)
>> >> >        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90)
>> >> >        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:166)
>> >> >
>> >> > And
>> >> >
>> >> > java.lang.OutOfMemoryError: Java heap space
>> >> >        at java.lang.StringCoding.encode(StringCoding.java:266)
>> >> >        at java.lang.StringCoding.encode(StringCoding.java:284)
>> >> >        at java.lang.String.getBytes(String.java:987)
>> >> >        at org.apache.cassandra.utils.FBUtilities.hash(FBUtilities.java:178)
>> >> >        at org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:116)
>> >> >        at org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:44)
>> >> >        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:148)
>> >> >        at org.apache.cassandra.db.Memtable.put(Memtable.java:143)
>> >> >        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:478)
>> >> >        at org.apache.cassandra.db.Table.apply(Table.java:445)
>> >> >        at org.apache.cassandra.db.CommitLog$3.run(CommitLog.java:365)
>> >> >        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> >        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> >        at java.lang.Thread.run(Thread.java:619)
>> >> >
>> >> > I upped the Xmx value from 4G to 6G and it seems to be doing okay,
however
>> >> > it seems odd that it can run mostly fine with 4G, but fail to restart
with
>> >> > that much memory.  Maybe this ticket's issue is back?
>> >> >
>> >> > https://issues.apache.org/jira/browse/CASSANDRA-609
>> >> >
>> >> > Anyway, I'm hoping thing will recover with 6G then I can restart again
with 4G and things will be good.
>> >> >
>> >> > I'd also like a better understanding of why cassandra might OOM in
general.
>> >> > Are there settings which minimize the chances of OOM?  This instance
has
>> >> > 2 column families and I have
>> >> >
>> >> >  <MemtableSizeInMB>512</MemtableSizeInMB>
>> >> >  <MemtableObjectCountInMillions>1.0</MemtableObjectCountInMillions>
>> >> >  <MemtableFlushAfterMinutes>1440</MemtableFlushAfterMinutes>
>> >> >
>> >> > So if I understand these settings, memtables can at most be 512MB in
size
>> >> > or consist of 1 million objects before they are flushed to disk.  The
maximum
>> >> > time before they will be flushed is 24 hours.  So does that mean if
I fill
>> >> > up 8G or 16 memtables in less than 24 hours, I've basically used all
the
>> >> > memory available to me?  I assume there are other things using memory,
>> >> > (indexes, etc), how is that limited?  Anyway, any information about
what
>> >> > is used where would be appreciated.
>> >> >
>> >> > Thanks,
>> >> >
>> >> > -Anthony
>> >> >
>> >> > --
>> >> > ------------------------------------------------------------------------
>> >> > Anthony Molinaro                           <anthonym@alumni.caltech.edu>
>> >> >
>> >
>> > --
>> > ------------------------------------------------------------------------
>> > Anthony Molinaro                           <anthonym@alumni.caltech.edu>
>> >
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym@alumni.caltech.edu>
>

Mime
View raw message