Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of comomore@gmail.com designates
 209.85.128.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGBp8g8BvEv_30DZWmwu=dY7X2JN3n4spJDVoe6meEe=trbwxw@mail.gmail.com>
References: 
 <CAGBp8g8BvEv_30DZWmwu=dY7X2JN3n4spJDVoe6meEe=trbwxw@mail.gmail.com>
Date: Thu, 19 Dec 2013 07:32:33 -0600
Message-ID: 
 <CAP7WDFX4u7o=OVxUDvmbfGS3B=jS1BmADNZyvgGXMjNoo=AaZw@mail.gmail.com>
Subject: Re: MUTATION messages dropped
From: srmore <comomore@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=047d7b6d99b0b139b904ede3313f

--047d7b6d99b0b139b904ede3313f
Content-Type: text/plain; charset=ISO-8859-1

What version of Cassandra are you running ? I used to see them a lot with
1.2.9, I could correlate the dropped messages with the heap usage almost
every time, so check in the logs whether you are getting GC'd. In this
respect 1.2.12 appears to be more stable. Moving to 1.2.12 took care of
this for us.

Thanks,
Sandeep


On Thu, Dec 19, 2013 at 6:12 AM, Alexander Shutyaev <shutyaev@gmail.com>wrote:

> Hi all!
>
> We've had a problem with cassandra recently. We had 2 one-minute periods
> when we got a lot of timeouts on the client side (the only timeouts during
> 9 days we are using cassandra in production). In the logs we've found
> corresponding messages saying something about MUTATION messages dropped.
>
> Now, the official faq [1] says that this is an indicator that the load is
> too high. We've checked our monitoring and found out that 1-minute average
> cpu load had a local peak at the time of the problem, but it was like 0.8
> against 0.2 usual which I guess is nothing for a 2 core virtual machine.
> We've also checked java threads - there was no peak there and their count
> was reasonable ~240-250.
>
> Can anyone give us a hint - what should we monitor to see this "high load"
> and what should we tune to make it acceptable?
>
> Thanks in advance,
> Alexander
>
> [1] http://wiki.apache.org/cassandra/FAQ#dropped_messages
>

--047d7b6d99b0b139b904ede3313f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>What version of Cassandra are you running ? I used to=
 see them a lot with 1.2.9, I could correlate the dropped messages with the=
 heap usage almost every time, so check in the logs whether you are getting=
 GC&#39;d. In this respect 1.2.12 appears to be more stable. Moving to 1.2.=
12 took care of this for us.<br>
<br></div>Thanks,<br>Sandeep<br><div><div><div><div class=3D"gmail_extra"><=
br><br><div class=3D"gmail_quote">On Thu, Dec 19, 2013 at 6:12 AM, Alexande=
r Shutyaev <span dir=3D"ltr">&lt;<a href=3D"mailto:shutyaev@gmail.com" targ=
et=3D"_blank">shutyaev@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi all!<div><br></div><div>=
We&#39;ve had a problem with cassandra recently. We had 2 one-minute period=
s when we got a lot of timeouts on the client side (the only timeouts durin=
g 9 days we are using cassandra in production). In the logs we&#39;ve found=
 corresponding messages saying something about MUTATION messages dropped.</=
div>

<div><br></div><div>Now, the official faq [1] says that this is an indicato=
r that the load is too high. We&#39;ve checked our monitoring and found out=
 that 1-minute average cpu load had a local peak at the time of the problem=
, but it was like 0.8 against 0.2 usual which I guess is nothing for a 2 co=
re virtual machine. We&#39;ve also checked java threads - there was no peak=
 there and their count was reasonable ~240-250.</div>

<div><br></div><div>Can anyone give us a hint - what should we monitor to s=
ee this &quot;high load&quot; and what should we tune to make it acceptable=
?</div><div><br></div><div>Thanks in advance,</div>
<div>Alexander</div><div><br></div><div>[1]=A0<a href=3D"http://wiki.apache=
.org/cassandra/FAQ#dropped_messages" target=3D"_blank">http://wiki.apache.o=
rg/cassandra/FAQ#dropped_messages</a></div></div>
</blockquote></div><br></div></div></div></div></div>

--047d7b6d99b0b139b904ede3313f--