incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jared Biel <jared.b...@bolderthinking.com>
Subject Re: MUTATION messages dropped
Date Fri, 20 Dec 2013 06:18:00 GMT
I can't comment on your specific issue, but I don't know if running 2.0.0
in production is a good idea. At the very least I'd try upgrading to the
latest 2.0.X (currently 2.0.3.)

 https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/

On 20 December 2013 06:08, Alexander Shutyaev <shutyaev@gmail.com> wrote:

> Thanks for you answers.
>
> *srmore*,
>
> We are using v2.0.0. As for GC I guess it does not correlate in our case,
> because we had cassandra running 9 days under production load and no
> dropped messages and I guess that during this time there were a lot of GCs.
>
> *Ken*,
>
> I've checked the values you indicated. Here they are:
>
> node1     6498
> node2     6476
> node3     6642
>
> I guess this is not good :) What can we do to fix this problem?
>
>
> 2013/12/19 Ken Hancock <ken.hancock@schange.com>
>
>> We had issues where the number of CF families that were being flushed
>> would align and then block writes for a very brief period. If that happened
>> when a bunch of writes came in, we'd see a spike in Mutation drops.
>>
>> Check nodetool tpstats for FlushWriter all time blocked.
>>
>>
>> On Thu, Dec 19, 2013 at 7:12 AM, Alexander Shutyaev <shutyaev@gmail.com>wrote:
>>
>>> Hi all!
>>>
>>> We've had a problem with cassandra recently. We had 2 one-minute periods
>>> when we got a lot of timeouts on the client side (the only timeouts during
>>> 9 days we are using cassandra in production). In the logs we've found
>>> corresponding messages saying something about MUTATION messages dropped.
>>>
>>> Now, the official faq [1] says that this is an indicator that the load
>>> is too high. We've checked our monitoring and found out that 1-minute
>>> average cpu load had a local peak at the time of the problem, but it was
>>> like 0.8 against 0.2 usual which I guess is nothing for a 2 core virtual
>>> machine. We've also checked java threads - there was no peak there and
>>> their count was reasonable ~240-250.
>>>
>>> Can anyone give us a hint - what should we monitor to see this "high
>>> load" and what should we tune to make it acceptable?
>>>
>>> Thanks in advance,
>>> Alexander
>>>
>>> [1] http://wiki.apache.org/cassandra/FAQ#dropped_messages
>>>
>>
>>
>>
>> --
>>  *Ken Hancock *| System Architect, Advanced Advertising
>> SeaChange International
>> 50 Nagog Park
>> Acton, Massachusetts 01720
>> ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC<http://www.schange.com/en-US/Company/InvestorRelations.aspx>
>>
>> Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hancock@schange.com
>>  | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
>> LinkedIn] <http://www.linkedin.com/in/kenhancock>
>>
>> [image: SeaChange International]
>>  <http://www.schange.com/>This e-mail and any attachments may contain
>> information which is SeaChange International confidential. The information
>> enclosed is intended only for the addressees herein and may not be copied
>> or forwarded without permission from SeaChange International.
>>
>
>

Mime
View raw message