incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: restarting node makes cpu load of the entire cluster to raise
Date Wed, 18 Jun 2014 12:05:13 GMT
A simple restart of a node with no changes give this result.

logs output : https://gist.github.com/arodrime/db9ab152071d1ad39f26

Here are some screenshot:

- htop from a node immediatly after restarting
- opscenter ring view (show load cpu on all nodes)
- opscenter dashboard shows the impact of a restart on latency (can affect
writes or reads, it depends, reaction seems to be quite random)


2014-06-18 13:35 GMT+02:00 Jonathan Lacefield <jlacefield@datastax.com>:

> Hello
>
>   Have you checked the log file to see what's happening during startup
> ?   What caused the rolling restart?  Did you preform an upgrade or
> change a config?
>
> > On Jun 18, 2014, at 5:40 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
> >
> > Hi guys
> >
> > Using 1.2.11, when I try to rolling restart the cluster, any node I
> restart makes the whole cluster cpu load to increase, reaching a "red"
> state in opscenter (load from 3-4 to 20+). This happens once the node is
> back online.
> >
> > The restarted node uses 100 % cpu for 5 - 10 min and sometimes drop
> mutations.
> >
> > I have tried to throttle handoff to 256 (instead of 1024), yet it
> doesn't seems to help that much.
> >
> > Disks are not the bottleneck. PARNEW GC increase a bit, but nothing
> problematic I think.
> >
> > Basically, what could be happening on node restart ? What is taking that
> much CPU on every machine ? There is no steal or iowait.
> >
> > What can I try to tune ?
> >
>

Mime
View raw message