incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Coli <rc...@eventbrite.com>
Subject Re: Nodes get stuck
Date Wed, 21 Aug 2013 17:47:17 GMT
On Tue, Aug 20, 2013 at 11:35 PM, Keith Wright <kwright@nanigans.com> wrote:

> Still looking for help!  We have stopped almost ALL traffic to the cluster
> and still some nodes are showing almost 1000% CPU for cassandra with no
> iostat activity.   We were running cleanup on one of the nodes that was not
> showing load spikes however now when I attempt to stop cleanup there via
> nodetool stop cleanup the java task for stopping cleanup itself is at 1500%
> and has not returned after 2 minutes.  This is VERY odd behavior.  Any
> ideas?  Hardware failure?  Network?  We are not seeing anything there but
> wanted to get ideas.
>

The most obvious answer is that somehow the problem nodes hit a magical
threshold which makes them "thrash" with GC.

If you restart the affected nodes, does the error  condition return? If so,
how quickly?

=Rob

Mime
View raw message