incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: OOM during restart
Date Tue, 21 Jun 2011 17:27:47 GMT
If you're OOMing on restart you WILL OOM during normal usage given
heavy enough write load.  Definitely adjust memtable thresholds down
or, as Dominic suggests, upgrade to 0.8.

On Tue, Jun 21, 2011 at 12:02 PM, Dominic Williams
<dwilliams@system7.co.uk> wrote:
> Hi gabe,
> What you need to do is the following:
> 1. Adjust cassandra.yaml so when this node is starting up it is not
> contacted by other nodes e.g. set thrift to 9061 and storage to 7001
> 2. Copy your commit logs to tmp sub-folder e.g. commitLog/tmp
> 3. Copy a small number of commit logs back into main commit log folder (be
> careful to copy the <id>.log and <id>.log.header file together)
> 4. Start up the node. When it has successfully started up, and therefore you
> know it has processed the commit logs, go back to step 3 and repeat
> 5. When you have no more commit logs remaining in tmp, you can revert
> cassandra.yaml and restart.. your node should be up again
> You might want to
> read http://ria101.wordpress.com/2011/02/08/cassandra-the-importance-of-system-clocks-avoiding-oom-and-how-to-escape-oom-meltdown/
> With Version 0.8 you can set a global memory threshold for the memtables so
> this kind of problem should become greatly reduced
> Best, Dominic
>
> On 20 June 2011 23:24, Gabriel Ki <gabeki@gmail.com> wrote:
>>
>> Hi,
>>
>> Cassandra: 7.6-2
>> I was restarting a node and ran into OOM while replaying the commit log.
>> I am not able to bring the node up again.
>>
>> DEBUG 15:11:43,501 forceFlush requested but everything is clean
>> <--------  For this I don't know what to do.
>> java.lang.OutOfMemoryError: Java heap space
>>     at
>> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:123)
>>     at
>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:395)
>>     at
>> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:76)
>>     at
>> org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:2238)
>>     at
>> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:166)
>>     at org.apache.cassandra.db.Memtable.access$000(Memtable.java:49)
>>     at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:189)
>>     at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>     at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>     at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>     at java.lang.Thread.run(Thread.java:662)
>>
>> Any help will be appreciated.
>>
>> If I update the schema while a node is down, the new schema is loaded
>> before the flushing when the node is brought up again, correct?
>>
>> Thanks,
>> -gabe
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message