incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Coli <rc...@eventbrite.com>
Subject Re: Nodes get stuck
Date Wed, 21 Aug 2013 17:48:30 GMT
On Wed, Aug 21, 2013 at 10:47 AM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Tue, Aug 20, 2013 at 11:35 PM, Keith Wright <kwright@nanigans.com>wrote:
>
>> Still looking for help!  We have stopped almost ALL traffic to the
>> cluster and still some nodes are showing almost 1000% CPU for cassandra
>> with no iostat activity.   We were running cleanup on one of the nodes that
>> was not showing load spikes however now when I attempt to stop cleanup
>> there via nodetool stop cleanup the java task for stopping cleanup itself
>> is at 1500% and has not returned after 2 minutes.  This is VERY odd
>> behavior.  Any ideas?  Hardware failure?  Network?  We are not seeing
>> anything there but wanted to get ideas.
>>
>
> The most obvious answer is that somehow the problem nodes hit a magical
> threshold which makes them "thrash" with GC.
>
> If you restart the affected nodes, does the error  condition return? If
> so, how quickly?
>

Lol, missed the rest of the responses on thread. NVMD. :D

=Rob

Mime
View raw message