incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Molinaro <antho...@alumni.caltech.edu>
Subject Re: OOM on restart
Date Sat, 13 Feb 2010 00:04:34 GMT
Okay, is it better written as

NumMemtables = 1 + 2 * AvailableProcessors + NumDataFileDirectory

Thus estimated maximum memory is

MemtableMemoryUsage = MemtableSizeInMB * NumMemtables

So in my case where I have 2 core machines with 3 datafile directories

MemtableMemoryUsage = 512 * (1 + 2 * 2 + 3) = 4096MB

which is basically greater than the 4G I've given to Xmx, thus the OOM?

-Anthony

On Fri, Feb 12, 2010 at 03:09:10PM -0600, Jonathan Ellis wrote:
> 0.5 allows 1 + 2 * Runtime.getRuntime().availableProcessors()
> Memtables + 1 per DataFileLocation to be waiting for flush before it
> will block writes (or log replay) to give those time to flush out.
> So, it sounds like you just need to lower your Memtable max
> size/object count.
> 
> On Fri, Feb 12, 2010 at 2:52 PM, Anthony Molinaro
> <anthonym@alumni.caltech.edu> wrote:
> > 0.5.0 final.  I was able to get things going again by upping the memory
> > then lowering it after a successful restart, but I would like to know how
> > to minimize the chances of OOM via tuning.
> >
> > -Anthony
> >
> > On Thu, Feb 11, 2010 at 01:29:16PM -0600, Jonathan Ellis wrote:
> >> What version are you on these days? :)
> >>
> >> On Thu, Feb 11, 2010 at 12:11 PM, Anthony Molinaro
> >> <anthonym@alumni.caltech.edu> wrote:
> >> > Hi,
> >> >
> >> >  I've been having nodes failing recently with OOM exceptions (not sure
> >> > why, but we have had an increase in traffic so that could be a cause).
> >> > Most nodes have restarted fine, one node however, has been having problems
> >> > restarting.  It was failing with
> >> >
> >> > java.lang.OutOfMemoryError: Java heap space
> >> >        at java.util.Arrays.copyOfRange(Arrays.java:3209)
> >> >        at java.lang.String.<init>(String.java:216)
> >> >        at java.io.DataInputStream.readUTF(DataInputStream.java:644)
> >> >        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
> >> >        at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:104)
> >> >        at org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:308)
> >> >        at org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:318)
> >> >        at org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:271)
> >> >        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:338)
> >> >        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65)
> >> >        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90)
> >> >        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:166)
> >> >
> >> > And
> >> >
> >> > java.lang.OutOfMemoryError: Java heap space
> >> >        at java.lang.StringCoding.encode(StringCoding.java:266)
> >> >        at java.lang.StringCoding.encode(StringCoding.java:284)
> >> >        at java.lang.String.getBytes(String.java:987)
> >> >        at org.apache.cassandra.utils.FBUtilities.hash(FBUtilities.java:178)
> >> >        at org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:116)
> >> >        at org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:44)
> >> >        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:148)
> >> >        at org.apache.cassandra.db.Memtable.put(Memtable.java:143)
> >> >        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:478)
> >> >        at org.apache.cassandra.db.Table.apply(Table.java:445)
> >> >        at org.apache.cassandra.db.CommitLog$3.run(CommitLog.java:365)
> >> >        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >        at java.lang.Thread.run(Thread.java:619)
> >> >
> >> > I upped the Xmx value from 4G to 6G and it seems to be doing okay, however
> >> > it seems odd that it can run mostly fine with 4G, but fail to restart with
> >> > that much memory.  Maybe this ticket's issue is back?
> >> >
> >> > https://issues.apache.org/jira/browse/CASSANDRA-609
> >> >
> >> > Anyway, I'm hoping thing will recover with 6G then I can restart again
with 4G and things will be good.
> >> >
> >> > I'd also like a better understanding of why cassandra might OOM in general.
> >> > Are there settings which minimize the chances of OOM?  This instance has
> >> > 2 column families and I have
> >> >
> >> >  <MemtableSizeInMB>512</MemtableSizeInMB>
> >> >  <MemtableObjectCountInMillions>1.0</MemtableObjectCountInMillions>
> >> >  <MemtableFlushAfterMinutes>1440</MemtableFlushAfterMinutes>
> >> >
> >> > So if I understand these settings, memtables can at most be 512MB in size
> >> > or consist of 1 million objects before they are flushed to disk.  The
maximum
> >> > time before they will be flushed is 24 hours.  So does that mean if I
fill
> >> > up 8G or 16 memtables in less than 24 hours, I've basically used all the
> >> > memory available to me?  I assume there are other things using memory,
> >> > (indexes, etc), how is that limited?  Anyway, any information about what
> >> > is used where would be appreciated.
> >> >
> >> > Thanks,
> >> >
> >> > -Anthony
> >> >
> >> > --
> >> > ------------------------------------------------------------------------
> >> > Anthony Molinaro                           <anthonym@alumni.caltech.edu>
> >> >
> >
> > --
> > ------------------------------------------------------------------------
> > Anthony Molinaro                           <anthonym@alumni.caltech.edu>
> >

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@alumni.caltech.edu>

Mime
View raw message