Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=to:from
	:subject:date:message-id:content-type:mime-version:in-reply-to;
	 q=dns; s=thelastpickle.com; b=dX3Lt/KFxyiuqmfqs4Hfh2gW0Ye9W8mXz
	9xzYlnSPYwetUUWHK9SGNasNmcFj0YyTmqnw0O/8Unu1/RVMWyWjwVdc0OZvU7Xr
	BHsgprAeoguHKAuIQMwsBClI8CR7m2ilWUpty7R0WVKuDLwtnE5nvdkJwBptLk5z
	dJ1pwhmwzU=
To: user@cassandra.apache.org
From: Aaron Morton <aaron@thelastpickle.com>
Subject: Re: JVM OOM on node startup
Date: Tue, 30 Nov 2010 20:41:10 GMT
Message-id: <1e385857-68eb-668d-ae91-958404432b43@me.com>
Content-Type: multipart/alternative;
 boundary=Apple-Webmail-42--f6ec6977-e60c-ca57-e12f-ccfba3c61765
MIME-Version: 1.0
In-Reply-To: <7D13E52A-CE61-41BE-9C87-8F27C726F8FC@grnoc.iu.edu>


--Apple-Webmail-42--f6ec6977-e60c-ca57-e12f-ccfba3c61765
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
    charset=ISO-8859-1;
    format=flowed

Looks like it's trying to load your row cache and running out of memory, p=
robably because you reduced the memory. The cassandra-env.sh script would =
have been giving it 2GB.=A01Gb heap is probably going to be to small.=A0=0A=
=0AWas this the same error you were getting before you reduced the memory =
?=A0=0A=0ATry deleting the caches, the path is specified by the saved_cach=
es_directory setting in cassandra.yaml.=A0=0A=0AAlso what version are you =
using ? The error=A0Caused by: javax.management.AttributeNotFoundException=
: No such attribute: ActiveCount reminds me of a problem in beta 1.=A0=0A=0A=
Hope that helps.=A0=0AAaron=0A=0AOn 01 Dec, 2010,at 09:28 AM, Brayton Thom=
pson <thompsbp@grnoc.iu.edu> wrote:=0A=0AHello again.=0AWe have 3 nodes an=
d were testing what happens when a node goes down. There is roughly 10gb o=
f data on each node. The node we "simulated" dieing was working just fine =
under the load. Then we killed it. The ring performed admirably, But upon =
restarting the node it dies every time of JVM OOM errors. I have forced a =
JVM heap size of 1024mb in the startup file. (did this because adaptive he=
ap size was causing oom errors with normal usage.) The machines are 2 core=
 4gb ram vm's.=0A=0AI've read the Riptano troubleshooting guide... http://=
www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-er=
rors But im not sure if these apply in this case since it is only dieing o=
n startup.=0A=0AHere is a link to the startup logs as it dies.=0Ahttp://pa=
stebin.com/BEXeVvCX=0A=0AThank you for any help you can provide.
--Apple-Webmail-42--f6ec6977-e60c-ca57-e12f-ccfba3c61765
Content-Type: multipart/related;
    type="text/html";
    boundary=Apple-Webmail-86--f6ec6977-e60c-ca57-e12f-ccfba3c61765


--Apple-Webmail-86--f6ec6977-e60c-ca57-e12f-ccfba3c61765
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
    charset=ISO-8859-1;

<html><body><div>Looks like it's trying to load your row cache and running out of memo=
ry, probably because you reduced the memory. The cassandra-env.sh script w=
ould have been giving it 2GB.&nbsp;1Gb heap is probably going to be to sma=
ll.&nbsp;</div><div><meta http-equiv=3D"content-type" content=3D"text/html=
; charset=3Dutf-8"></div><div><br></div><div>Was this the same error you w=
ere getting before you reduced the memory ?&nbsp;</div><div><br></div><div=
>Try deleting the caches, the path is specified by the saved_caches_direct=
ory setting in cassandra.yaml.&nbsp;</div><div><br></div><div>Also what ve=
rsion are you using ? The error&nbsp;<span class=3D"Apple-style-span" styl=
e=3D"font-family: monospace; font-size: 12px; color: rgb(24, 24, 24); line=
-height: 18px; ">Caused by: javax.management.AttributeNotFoundException: N=
o such attribute: ActiveCount reminds me of a problem in beta 1.&nbsp;</sp=
an></div><div><span class=3D"Apple-style-span" style=3D"font-family: monos=
pace; font-size: 12px; color: rgb(24, 24, 24); line-height: 18px; "><br></=
span></div><div><meta http-equiv=3D"content-type" content=3D"text/html; ch=
arset=3Dutf-8"></div><div>Hope that helps.&nbsp;</div><div>Aaron</div><div=
><br>On 01 Dec, 2010,at 09:28 AM, Brayton Thompson &lt;thompsbp@grnoc.iu.e=
du&gt; wrote:<br><br></div><div><blockquote type=3D"cite"><div><div class=3D=
"_stretch">Hello again.<br>=0A	We have 3 nodes and were testing what happe=
ns when a node goes down. There is roughly 10gb of data on each node. The =
node we "simulated" dieing was working just fine under the load. Then we k=
illed it. The ring performed admirably, But upon restarting the node it di=
es every time of JVM OOM errors.  I have forced a JVM heap size of 1024mb =
in the startup file. (did this because adaptive heap size was causing oom =
errors with normal usage.) The machines are 2 core 4gb ram vm's.<br>=0A<br=
>=0AI've read the Riptano troubleshooting guide... <a href=3D"http://www.r=
iptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors"=
 _mce_href=3D"http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-=
are-dying-with-oom-errors">http://www.riptano.com/docs/0.6/troubleshooting=
/index#nodes-are-dying-with-oom-errors</a> But im not sure if these apply =
in this case since it is only dieing on startup.<br>=0A<br>=0AHere is a li=
nk to the startup logs as it dies.<br>=0A<a href=3D"http://pastebin.com/BE=
XeVvCX" _mce_href=3D"http://pastebin.com/BEXeVvCX">http://pastebin.com/BEX=
eVvCX</a><br>=0A<br>=0AThank you for any help you can provide.</div></div>=
</blockquote></div></body></html>
--Apple-Webmail-86--f6ec6977-e60c-ca57-e12f-ccfba3c61765--

--Apple-Webmail-42--f6ec6977-e60c-ca57-e12f-ccfba3c61765--