cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: OOM on restart
Date Thu, 11 Feb 2010 19:29:16 GMT
What version are you on these days? :)

On Thu, Feb 11, 2010 at 12:11 PM, Anthony Molinaro
<anthonym@alumni.caltech.edu> wrote:
> Hi,
>
>  I've been having nodes failing recently with OOM exceptions (not sure
> why, but we have had an increase in traffic so that could be a cause).
> Most nodes have restarted fine, one node however, has been having problems
> restarting.  It was failing with
>
> java.lang.OutOfMemoryError: Java heap space
>        at java.util.Arrays.copyOfRange(Arrays.java:3209)
>        at java.lang.String.<init>(String.java:216)
>        at java.io.DataInputStream.readUTF(DataInputStream.java:644)
>        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>        at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:104)
>        at org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:308)
>        at org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:318)
>        at org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:271)
>        at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:338)
>        at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65)
>        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90)
>        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:166)
>
> And
>
> java.lang.OutOfMemoryError: Java heap space
>        at java.lang.StringCoding.encode(StringCoding.java:266)
>        at java.lang.StringCoding.encode(StringCoding.java:284)
>        at java.lang.String.getBytes(String.java:987)
>        at org.apache.cassandra.utils.FBUtilities.hash(FBUtilities.java:178)
>        at org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:116)
>        at org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:44)
>        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:148)
>        at org.apache.cassandra.db.Memtable.put(Memtable.java:143)
>        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:478)
>        at org.apache.cassandra.db.Table.apply(Table.java:445)
>        at org.apache.cassandra.db.CommitLog$3.run(CommitLog.java:365)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
>
> I upped the Xmx value from 4G to 6G and it seems to be doing okay, however
> it seems odd that it can run mostly fine with 4G, but fail to restart with
> that much memory.  Maybe this ticket's issue is back?
>
> https://issues.apache.org/jira/browse/CASSANDRA-609
>
> Anyway, I'm hoping thing will recover with 6G then I can restart again with 4G and things
will be good.
>
> I'd also like a better understanding of why cassandra might OOM in general.
> Are there settings which minimize the chances of OOM?  This instance has
> 2 column families and I have
>
>  <MemtableSizeInMB>512</MemtableSizeInMB>
>  <MemtableObjectCountInMillions>1.0</MemtableObjectCountInMillions>
>  <MemtableFlushAfterMinutes>1440</MemtableFlushAfterMinutes>
>
> So if I understand these settings, memtables can at most be 512MB in size
> or consist of 1 million objects before they are flushed to disk.  The maximum
> time before they will be flushed is 24 hours.  So does that mean if I fill
> up 8G or 16 memtables in less than 24 hours, I've basically used all the
> memory available to me?  I assume there are other things using memory,
> (indexes, etc), how is that limited?  Anyway, any information about what
> is used where would be appreciated.
>
> Thanks,
>
> -Anthony
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym@alumni.caltech.edu>
>

Mime
View raw message