Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	thelastpickle.com; b=3RWH1XIbM1QJPcqNEPGJRSleJAUHaNp73h11D7rGGWk
	iuVAUPuC0d1aC8bh5CeVM+2Qo+Fg/yDXJK5OXaBWjI0SDLRJ6cHj7a9SY8bvsfVC
	dlWrLsr779VnomZ9uGzmWuTS74DIBHs1iv/M8/ud+OdTVKKbV9G/KudZWZBI4poQ
	=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: OOM during restart
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: <BANLkTinT2VEf8hc13EeiJks_VXgMjFGJOA@mail.gmail.com>
Date: Tue, 21 Jun 2011 23:40:21 +1200
Content-Transfer-Encoding: quoted-printable
Message-Id: <F054D7EB-310C-4E66-9B7C-F21C0EF99997@thelastpickle.com>
References: <BANLkTinT2VEf8hc13EeiJks_VXgMjFGJOA@mail.gmail.com>
To: user@cassandra.apache.org

AFAIK the node will not announce itself in the ring until the log replay =
is complete, so it will not get the schema update until after log =
replay. If possible i'd avoid making the schema change until you have =
solved this problem.

My theory on OOM during log replay is that the high speed inserts are a =
good way of finding out if the maximum memory required by the schema is =
too big to fit in the JVM. How big is the max JVM Heap SIze and do you =
have a lot of CF's?

The simple solution it to either (temporarily) increase the JVM Heap =
Size or move the log files so that the server can process only one at a =
time. The JVM option D.cassandra_ring=3Dfalse will stop the node from =
joining the cluster and stop other nodes sending requests to it until =
you have sorted it out.=20

Hope that helps.=20
 =20
=20
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 21 Jun 2011, at 10:24, Gabriel Ki wrote:

> Hi,
>=20
> Cassandra: 7.6-2
> I was restarting a node and ran into OOM while replaying the commit =
log.  I am not able to bring the node up again.
>=20
> DEBUG 15:11:43,501 forceFlush requested but everything is clean      =
<--------  For this I don't know what to do.
> java.lang.OutOfMemoryError: Java heap space
>     at =
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRando=
mAccessFile.java:123)
>     at =
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWr=
iter.java:395)
>     at =
org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:76=
)
>     at =
org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilySt=
ore.java:2238)
>     at =
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:166)
>     at org.apache.cassandra.db.Memtable.access$000(Memtable.java:49)
>     at =
org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:189)
>     at =
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>     at =
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.=
java:886)
>     at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:908)
>     at java.lang.Thread.run(Thread.java:662)
>=20
> Any help will be appreciated.  =20
>=20
> If I update the schema while a node is down, the new schema is loaded =
before the flushing when the node is brought up again, correct? =20
>=20
> Thanks,
> -gabe