Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of rcoli@eventbrite.com
 designates 209.85.215.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <CF166144.D2220%rwille@fold3.com>
References: 
 <CAKmMYa9uVUS+QvED=BQdjJWMCcRBOQ5ewM7TmYz1OB2e-MyV8Q@mail.gmail.com>
	<CF166144.D2220%rwille@fold3.com>
Date: Tue, 4 Feb 2014 16:15:44 -0800
Message-ID: 
 <CAEDUwd0wukyxzxVfXi83R9EFgm24HRRE2DvqND0=AF5XQCXM8g@mail.gmail.com>
Subject: Re: Lots of deletions results in death by GC
From: Robert Coli <rcoli@eventbrite.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Content-Type: multipart/alternative; boundary=001a11c3733e6d6dc704f19da8de

--001a11c3733e6d6dc704f19da8de
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Feb 4, 2014 at 4:10 PM, Robert Wille <rwille@fold3.com> wrote:

> I would almost think that this is caused by high load on the server, but
> I've never seen CPU utilization go above about two of my eight available
> cores. If high load triggers this problem, then that is very disconcerting.
> That means that a CPU spike could permanently cripple a node. Okay, not
> permanently, but until a manual flush occurs.
>

While it is unlikely to be related to CPU (i/o much more likely...), the
JVM is certainly capable of finding itself in a low-heap situation which :

1) performs very poorly
2) does not crash
3) and cannot be recovered from

This usually requires a restart of the JVM. This mental model should be an
expectation for any application running in the JVM.

=Rob

--001a11c3733e6d6dc704f19da8de
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On T=
ue, Feb 4, 2014 at 4:10 PM, Robert Wille <span dir=3D"ltr">&lt;<a href=3D"m=
ailto:rwille@fold3.com" target=3D"_blank">rwille@fold3.com</a>&gt;</span> w=
rote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div style=3D"font-size:14px;font-family:Cal=
ibri,sans-serif;word-wrap:break-word"><div><span style=3D"font-size:14px">I=
 would almost think that this is caused by high load on the server, but I&r=
squo;ve never seen CPU utilization go above about two of my eight available=
 cores. If high load triggers this problem, then that is very disconcerting=
. That means that a CPU spike could permanently cripple a node. Okay, not p=
ermanently, but until a manual flush occurs.</span></div>
</div></blockquote><div><br></div><div>While it is unlikely to be related t=
o CPU (i/o much more likely...), the JVM is certainly capable of finding it=
self in a low-heap situation which :</div><div><br></div><div>1) performs v=
ery poorly</div>
<div>2) does not crash</div><div>3) and cannot be recovered from</div><div>=
<br></div><div>This usually requires a restart of the JVM. This mental mode=
l should be an expectation for any application running in the JVM.</div>
<div><br></div><div>=3DRob</div><div>&nbsp;</div></div></div></div>

--001a11c3733e6d6dc704f19da8de--