Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Michael Theroux <mtheroux2@yahoo.com>
Mime-Version: 1.0 (Apple Message framework v1283)
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_4B1362DA-6D64-4818-900D-1BB56E7ABA13"
Subject: Re: Really odd issue (AWS related?)
Date: Fri, 26 Apr 2013 10:14:23 -0400
In-Reply-To: 
 <CAHO4itzg2Dt=BboNbVw57TiFONwCCJHziShFExhwN+snUprQ=A@mail.gmail.com>
To: user@cassandra.apache.org
References: <DE800C10-9ABB-4D93-AA0C-12069C549613@yahoo.com>
 <CAEDUwd2RqdrjHLDEU58QcUbRi5ifY-2W9DH1Kuft3amro=K_wQ@mail.gmail.com>
 <39D73F09-23D5-45A5-8DEF-B45BBDFC4BE8@thelastpickle.com>
 <C2010779-AE31-4A57-B09C-C03E6C17DF22@yahoo.com>
 <CAHO4itzg2Dt=BboNbVw57TiFONwCCJHziShFExhwN+snUprQ=A@mail.gmail.com>
Message-Id: <64F68A8D-C846-47C5-AFC4-DBE047680EFB@yahoo.com>


--Apple-Mail=_4B1362DA-6D64-4818-900D-1BB56E7ABA13
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1

Thanks.

We weren't monitoring this value when the issue occurred, and this =
particular issue has not appeared for a couple of days (knock on wood).  =
Will keep an eye out though,

-Mike

On Apr 26, 2013, at 5:32 AM, Jason Wee wrote:

> top command? st : time stolen from this vm by the hypervisor
>=20
> jason
>=20
>=20
> On Fri, Apr 26, 2013 at 9:54 AM, Michael Theroux <mtheroux2@yahoo.com> =
wrote:
> Sorry, Not sure what CPU steal is :)
>=20
> I have AWS console with detailed monitoring enabled... things seem to =
track close to the minute, so I can see the CPU load go to 0... then =
jump at about the minute Cassandra reports the dropped messages,
>=20
> -Mike
>=20
> On Apr 25, 2013, at 9:50 PM, aaron morton wrote:
>=20
>>> The messages appear right after the node "wakes up".
>> Are you tracking CPU steal ?=20
>>=20
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Consultant
>> New Zealand
>>=20
>> @aaronmorton
>> http://www.thelastpickle.com
>>=20
>> On 25/04/2013, at 4:15 AM, Robert Coli <rcoli@eventbrite.com> wrote:
>>=20
>>> On Wed, Apr 24, 2013 at 5:03 AM, Michael Theroux =
<mtheroux2@yahoo.com> wrote:
>>>> Another related question.  Once we see messages being dropped on =
one node, our cassandra client appears to see this, reporting errors.  =
We use LOCAL_QUORUM with a RF of 3 on all queries.  Any idea why clients =
would see an error?  If only one node reports an error, shouldn't the =
consistency level prevent the client from seeing an issue?
>>>=20
>>> If the client is talking to a broken/degraded coordinator node, =
RF/CL
>>> are unable to protect it from RPCTimeout. If it is unable to
>>> coordinate the request in a timely fashion, your clients will get
>>> errors.
>>>=20
>>> =3DRob
>>=20
>=20
>=20


--Apple-Mail=_4B1362DA-6D64-4818-900D-1BB56E7ABA13
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=iso-8859-1

<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Thanks.<div><br></div><div>We weren't monitoring this value when the issue occurred, and this particular issue has not appeared for a couple of days (knock on wood). &nbsp;Will keep an eye out though,</div><div><br></div><div>-Mike</div><div><br><div><div>On Apr 26, 2013, at 5:32 AM, Jason Wee wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr"><div>top command? st : time stolen from this vm by the hypervisor</div><div><br></div><div style="">jason</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Apr 26, 2013 at 9:54 AM, Michael Theroux <span dir="ltr">&lt;<a href="mailto:mtheroux2@yahoo.com" target="_blank">mtheroux2@yahoo.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Sorry, Not sure what CPU steal is :)<div><br></div><div>I have AWS console with detailed monitoring enabled... things seem to track close to the minute, so I can see the CPU load go to 0... then jump at about the minute Cassandra reports the dropped messages,</div>
<div><br></div><div>-Mike</div><div><div class="h5"><div><br><div><div>On Apr 25, 2013, at 9:50 PM, aaron morton wrote:</div><br><blockquote type="cite"><div style="word-wrap:break-word"><blockquote type="cite">The messages appear right after the node "wakes up".<br>
</blockquote>Are you tracking CPU steal ?&nbsp;<div><br><div>
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-wrap:break-word;word-spacing:0px">
<div style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-wrap:break-word;word-spacing:0px">
<span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div style="word-wrap:break-word">
<span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><div style="word-wrap:break-word">
<div>-----------------</div><div>Aaron Morton</div><div>Freelance Cassandra Consultant</div><div>New Zealand</div><div><br></div><div>@aaronmorton</div><div><a href="http://www.thelastpickle.com/" target="_blank">http://www.thelastpickle.com</a></div>
</div></span></div></span></div></span></div></span></div></div>
</div>
<br><div><div>On 25/04/2013, at 4:15 AM, Robert Coli &lt;<a href="mailto:rcoli@eventbrite.com" target="_blank">rcoli@eventbrite.com</a>&gt; wrote:</div><br><blockquote type="cite">On Wed, Apr 24, 2013 at 5:03 AM, Michael Theroux &lt;<a href="mailto:mtheroux2@yahoo.com" target="_blank">mtheroux2@yahoo.com</a>&gt; wrote:<br>
<blockquote type="cite">Another related question. &nbsp;Once we see messages being dropped on one node, our cassandra client appears to see this, reporting errors. &nbsp;We use LOCAL_QUORUM with a RF of 3 on all queries. &nbsp;Any idea why clients would see an error? &nbsp;If only one node reports an error, shouldn't the consistency level prevent the client from seeing an issue?<br>
</blockquote><br>If the client is talking to a broken/degraded coordinator node, RF/CL<br>are unable to protect it from RPCTimeout. If it is unable to<br>coordinate the request in a timely fashion, your clients will get<br>
errors.<br><br>=Rob<br></blockquote></div><br></div></div></blockquote></div><br></div></div></div></div></blockquote></div><br></div>
</blockquote></div><br></div></body></html>
--Apple-Mail=_4B1362DA-6D64-4818-900D-1BB56E7ABA13--