incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: restarting node makes cpu load of the entire cluster to raise
Date Wed, 18 Jun 2014 12:43:06 GMT
This last command was supposed to be a best practice a few years ago, hope
it is still the case. I just added the recent "nodetool disablebinary"
part...


2014-06-18 14:36 GMT+02:00 Alain RODRIGUEZ <arodrime@gmail.com>:

> Thanks a lot for taking time to check the log.
>
> We just switched from 400M to 1600M NEW size in the cassandra-env.sh. It
> reduced our latency and the PARNEW GC time / second significantly...
> (described here
> http://tech.shift.com/post/74311817513/cassandra-tuning-the-jvm-for-read-heavy-workloads
> )
>
> Even when we had 400M the restart was behaving this way.
>
> We stop the node using : nodetool disablegossip && nodetool disablethrift
> && nodetool disablebinary && sleep 10 && nodetool drain &&
sleep 30 &&
> service cassandra stop
>
>
> 2014-06-18 14:23 GMT+02:00 Jonathan Lacefield <jlacefield@datastax.com>:
>
> There are several long Parnew pauses that were recorded during startup.
>>  The young gen size looks large too, if I am reading that line correctly.
>>  Did you happen to overwrite the default settings for MAX_HEAP and/or NEW
>> size in the cassandra-env.sh?  The large you gen size, set via the env.sh
>> file, could be causing longer than typical pauses, which could make your
>> node appear to be unresponsive and have high CPU (CPU for the ParNew GC
>> event).
>>
>> Check out this one - INFO 11:42:51,939 GC for ParNew: 2148 ms for 2
>> collections, 1256307568 used; max is 8422162432
>> That is a 2 second GC pause.  That's very high for ParNew.  We typically
>> want a lot of tiny ParNew events as opposed to large, and less frequent,
>> ParNew events.
>>
>> One other thing that was noticed, was that the node had a lot of log
>> segment replay's during startup.  You could avoid these, or minimize them,
>> by preforming a flush or drain before stopping and starting Cassandra.
>>  This will flush memtables and clear your log segments.
>>
>>
>>
>> Jonathan Lacefield
>> Solutions Architect, DataStax
>> (404) 822 3487
>>  <http://www.linkedin.com/in/jlacefield>
>>
>> <http://www.datastax.com/cassandrasummit14>
>>
>>
>>
>> On Wed, Jun 18, 2014 at 8:05 AM, Alain RODRIGUEZ <arodrime@gmail.com>
>> wrote:
>>
>>> A simple restart of a node with no changes give this result.
>>>
>>> logs output : https://gist.github.com/arodrime/db9ab152071d1ad39f26
>>>
>>> Here are some screenshot:
>>>
>>> - htop from a node immediatly after restarting
>>> - opscenter ring view (show load cpu on all nodes)
>>> - opscenter dashboard shows the impact of a restart on latency (can
>>> affect writes or reads, it depends, reaction seems to be quite random)
>>>
>>>
>>> 2014-06-18 13:35 GMT+02:00 Jonathan Lacefield <jlacefield@datastax.com>:
>>>
>>> Hello
>>>>
>>>>   Have you checked the log file to see what's happening during startup
>>>> ?   What caused the rolling restart?  Did you preform an upgrade or
>>>> change a config?
>>>>
>>>> > On Jun 18, 2014, at 5:40 AM, Alain RODRIGUEZ <arodrime@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi guys
>>>> >
>>>> > Using 1.2.11, when I try to rolling restart the cluster, any node I
>>>> restart makes the whole cluster cpu load to increase, reaching a "red"
>>>> state in opscenter (load from 3-4 to 20+). This happens once the node is
>>>> back online.
>>>> >
>>>> > The restarted node uses 100 % cpu for 5 - 10 min and sometimes drop
>>>> mutations.
>>>> >
>>>> > I have tried to throttle handoff to 256 (instead of 1024), yet it
>>>> doesn't seems to help that much.
>>>> >
>>>> > Disks are not the bottleneck. PARNEW GC increase a bit, but nothing
>>>> problematic I think.
>>>> >
>>>> > Basically, what could be happening on node restart ? What is taking
>>>> that much CPU on every machine ? There is no steal or iowait.
>>>> >
>>>> > What can I try to tune ?
>>>> >
>>>>
>>>
>>>
>>
>

Mime
View raw message