cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jai Bheemsen Rao Dhanwada <jaibheem...@gmail.com>
Subject Re: nodetool removenode causing the schema out of sync
Date Thu, 13 Jul 2017 19:19:28 GMT
Also changing the compaction throughput on the fly while removing nodes is
not scalable as we have 100s of nodes.
I can try and test though

On Thursday, July 13, 2017, Jai Bheemsen Rao Dhanwada <jaibheemsen@gmail.com>
wrote:

> Yes i did removenode and removenode force, and ecnountered same issue in
> both the cases.
>
> On Thursday, July 13, 2017, Subroto Barua <sbarua116@yahoo.com.invalid>
> wrote:
>
>> set streamthroughput higher than 200 on the source side and lower on the
>> target node
>>
>> just curious, have you tried removenode force?
>>
>>
>> On Thursday, July 13, 2017, 8:35:38 AM PDT, Jai Bheemsen Rao Dhanwada <
>> jaibheemsen@gmail.com> wrote:
>>
>>
>> Thank you Sean,
>>
>> you mean setstreamthroughput to a lower value on the node where we are
>> doing a "nodetool removenode <hostid>"?
>>
>>
>> On Thu, Jul 13, 2017 at 8:07 AM, Durity, Sean R <
>> SEAN_R_DURITY@homedepot.com> wrote:
>>
>> Late to this party, but Jeff is talking about nodetool
>> setstreamthroughput. The default in most versions is 200 Mb/s (set in yaml
>> file as stream_throughput_outbound_ megabits_per_sec). This is outbound
>> throttle only. So, if streams from multiple nodes are going to one, it can
>> get inundated.
>>
>>
>>
>> The nodetool command lets you change this on the fly (no bounce
>> required), but I don’t think it affects any current streaming from that
>> node (only future). You can use nodetool getstreamthroughput to see the
>> current value.
>>
>>
>>
>>
>>
>> Sean Durity
>>
>>
>>
>> *From:* Jai Bheemsen Rao Dhanwada [mailto:jaibheemsen@gmail.com]
>> *Sent:* Thursday, June 29, 2017 6:39 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: nodetool removenode causing the schema out of sync
>>
>>
>>
>> Thanks Jeff,
>>
>>
>>
>> Can you please suggest what value to tweak from the Cassandra side?
>>
>>
>>
>> On Thu, Jun 29, 2017 at 2:53 PM, Jeff Jirsa <jjirsa@apache.org> wrote:
>>
>>
>>
>> On 2017-06-29 13:45 (-0700), Jai Bheemsen Rao Dhanwada <
>> jaibheemsen@gmail.com> wrote:
>> > Hello Jeff,
>> >
>> > Sorry the Version I am using 2.1.16, my first email had typo.
>> > When I say schema out of sync
>> >
>> > 1. nodetool descriebcluster shows Schema versions same for all nodes.
>>
>> Ok got it, this is what I was most concerned with.
>>
>> > 2. nodetool removenode, shows the node down messages in the logs
>> > 3. nodetool describecluster during this 1-2 mins shows several nodes as
>> > UNREACHABLE and recovers with in a minute or two.
>>
>> This is likely due to overhead of streaming - you're probably running
>> pretty close to your tipping point, and your streaming throughput creates
>> enough GC pressure on the destinations to make them flap a bit. If you use
>> the streaming throughput throttle, you may be able to help mitigate that
>> somewhat (at the cost of speed).
>>
>>
>>
>>
>> ------------------------------ ------------------------------ ---------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra. apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>>
>> ------------------------------
>>
>> The information in this Internet Email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this Email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful. When addressed
>> to our clients any opinions or advice contained in this Email are subject
>> to the terms and conditions expressed in any applicable governing The Home
>> Depot terms of business or client engagement letter. The Home Depot
>> disclaims all responsibility and liability for the accuracy and content of
>> this attachment and for any damages or losses arising from any
>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>> items of a destructive nature, which may be contained in this attachment
>> and shall not be liable for direct, indirect, consequential or special
>> damages in connection with this e-mail message or its attachment.
>>
>>
>>

Mime
View raw message