cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Subroto Barua <sbarua...@yahoo.com.INVALID>
Subject Re: nodetool removenode causing the schema out of sync
Date Thu, 13 Jul 2017 15:59:30 GMT
set streamthroughput higher than 200 on the source side and lower on the target node
just curious, have you tried removenode force?

On Thursday, July 13, 2017, 8:35:38 AM PDT, Jai Bheemsen Rao Dhanwada <jaibheemsen@gmail.com>
wrote:

Thank you Sean,
you mean setstreamthroughput to a lower value on the node where we are doing a "nodetool removenode
<hostid>"?

On Thu, Jul 13, 2017 at 8:07 AM, Durity, Sean R <SEAN_R_DURITY@homedepot.com> wrote:


Late to this party, but Jeff is talking about nodetool setstreamthroughput. The default in
most versions is 200 Mb/s (set in yaml file as stream_throughput_outbound_ megabits_per_sec).
This is outbound throttle only. So, if streams from multiple nodes are going to one, it can
get inundated.

 

The nodetool command lets you change this on the fly (no bounce required), but I don’t think
it affects any current streaming from that node (only future). You can use nodetool getstreamthroughput
to see the current value.

 

 

Sean Durity

 

From: Jai Bheemsen Rao Dhanwada [mailto:jaibheemsen@gmail.com]
Sent: Thursday, June 29, 2017 6:39 PM
To: user@cassandra.apache.org
Subject: Re: nodetool removenode causing the schema out of sync

 

Thanks Jeff,

 

Can you please suggest what value to tweak from the Cassandra side?

 

On Thu, Jun 29, 2017 at 2:53 PM, Jeff Jirsa <jjirsa@apache.org> wrote:




On 2017-06-29 13:45 (-0700), Jai Bheemsen Rao Dhanwada <jaibheemsen@gmail.com> wrote:
> Hello Jeff,
>
> Sorry the Version I am using 2.1.16, my first email had typo.
> When I say schema out of sync
>
> 1. nodetool descriebcluster shows Schema versions same for all nodes.

Ok got it, this is what I was most concerned with.

> 2. nodetool removenode, shows the node down messages in the logs
> 3. nodetool describecluster during this 1-2 mins shows several nodes as
> UNREACHABLE and recovers with in a minute or two.

This is likely due to overhead of streaming - you're probably running pretty close to your
tipping point, and your streaming throughput creates enough GC pressure on the destinations
to make them flap a bit. If you use the streaming throughput throttle, you may be able to
help mitigate that somewhat (at the cost of speed).




------------------------------ ------------------------------ ---------
To unsubscribe, e-mail: user-unsubscribe@cassandra. apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


 


The information in this Internet Email is confidential and may be legally privileged. It is
intended solely for the addressee. Access to this Email by anyone else is unauthorized. If
you are not the intended recipient, any disclosure, copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject to the terms and
conditions expressed in any applicable governing The Home Depot terms of business or client
engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy
and content of this attachment and for any damages or losses arising from any inaccuracies,
errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature,
which may be contained in this attachment and shall not be liable for direct, indirect, consequential
or special damages in connection with this e-mail message or its attachment.



Mime
View raw message