Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of narendra.sharma@gmail.com
 designates 209.85.223.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAEDUwd0KQ+8NQRAoRWbdR8Vy-MgWj+qV6Zvav6AcrHgxdD0JpA@mail.gmail.com>
References: 
 <CAHamXHE0395XTAnOp6y3EwEnO+z3rjH99Jv6wVqr2fRJEaAHGQ@mail.gmail.com>
	<CAHamXHGDAp5TXRkT=0K=1GJATErNAe4f8nZFJFVv=8fUao_9fA@mail.gmail.com>
	<CAEDUwd0KQ+8NQRAoRWbdR8Vy-MgWj+qV6Zvav6AcrHgxdD0JpA@mail.gmail.com>
Date: Thu, 2 Jan 2014 21:00:05 -0800
Message-ID: 
 <CAHamXHEdTqD24xCa=mYBRWqPNOy-2ydFFyrqzNFvvpUijGezAQ@mail.gmail.com>
Subject: Re: Cassandra 1.1.6 crash without any exception or error in log
From: Narendra Sharma <narendra.sharma@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a11c2fafe941c7604ef09c8ac

--001a11c2fafe941c7604ef09c8ac
Content-Type: text/plain; charset=ISO-8859-1

The root cause turned out to be high heap. The Linux OOM Killer (
http://linux-mm.org/OOM_Killer) killed the process. It took some time to
figure out but very interesting. We knew high heap is a problem but had no
clue when the actual heap usage was well within limit and the process
disappeared. syslog helped figure this out.

About Linux OOM Killer
"It is the job of the linux 'oom killer' to *sacrifice* one or more
processes in order to free up memory for the system when all else fails"


On Thu, Jan 2, 2014 at 10:38 AM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Thu, Jan 2, 2014 at 8:13 AM, Narendra Sharma <narendra.sharma@gmail.com
> > wrote:
>
>> 8 node cluster running in aws. Any pointers where I should start looking?
>> No kill -9 in history.
>>
> You should start looking at instructions as to how to upgrade to at least
> the top of the 1.1 line... :D
>
> =Rob
>


-- 
Narendra Sharma
Software Engineer
*http://www.aeris.com <http://www.aeris.com>*
*http://narendrasharma.blogspot.com/ <http://narendrasharma.blogspot.com/>*

--001a11c2fafe941c7604ef09c8ac
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">The root cause turned out to be high heap. The Linux OOM K=
iller (<a href=3D"http://linux-mm.org/OOM_Killer">http://linux-mm.org/OOM_K=
iller</a>) killed the process. It took some time to figure out but very int=
eresting. We knew high heap is a problem but had no clue when the actual he=
ap usage was well within limit and the process disappeared. syslog helped f=
igure this out.<div>
<br></div><div>About Linux OOM Killer</div><div>&quot;<span style=3D"color:=
rgb(0,0,0);font-family:Arial,&#39;Lucida Grande&#39;,sans-serif;line-height=
:15px"><span style=3D"font-size:12px">It is the job of the linux &#39;oom k=
iller&#39; to </span><b><font size=3D"4">sacrifice</font></b><span style=3D=
"font-size:12px"> one or more processes in order to free up memory for the =
system when all else fails&quot;</span></span></div>
</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Thu,=
 Jan 2, 2014 at 10:38 AM, Robert Coli <span dir=3D"ltr">&lt;<a href=3D"mail=
to:rcoli@eventbrite.com" target=3D"_blank">rcoli@eventbrite.com</a>&gt;</sp=
an> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_extra">=
<div class=3D"im"><div class=3D"gmail_quote">On Thu, Jan 2, 2014 at 8:13 AM=
, Narendra Sharma <span dir=3D"ltr">&lt;<a href=3D"mailto:narendra.sharma@g=
mail.com" target=3D"_blank">narendra.sharma@gmail.com</a>&gt;</span> wrote:=
<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><p dir=3D"ltr">8 node cluster running in aws=
. Any pointers where I should start looking?<br>
No kill -9 in history.</p>
</blockquote></div></div>You should start looking at instructions as to how=
 to upgrade to at least the top of the 1.1 line... :D</div><div class=3D"gm=
ail_extra"><br></div><div class=3D"gmail_extra">=3DRob</div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><font color=
=3D"#999999">Narendra Sharma</font><div><font color=3D"#999999">Software En=
gineer</font><br><div><i><a href=3D"http://www.aeris.com" target=3D"_blank"=
>http://www.aeris.com</a></i></div>
<div><i><a href=3D"http://narendrasharma.blogspot.com/" target=3D"_blank">h=
ttp://narendrasharma.blogspot.com/</a></i></div><div><br></div></div>
</div>

--001a11c2fafe941c7604ef09c8ac--