Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 60BE0200CC4 for ; Thu, 13 Jul 2017 17:59:46 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5FC9616C466; Thu, 13 Jul 2017 15:59:46 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 55C3916C460 for ; Thu, 13 Jul 2017 17:59:45 +0200 (CEST) Received: (qmail 7690 invoked by uid 500); 13 Jul 2017 15:59:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 7680 invoked by uid 99); 13 Jul 2017 15:59:42 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jul 2017 15:59:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 64A2C183A50 for ; Thu, 13 Jul 2017 15:59:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.724 X-Spam-Level: **** X-Spam-Status: No, score=4.724 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FORGED_MUA_MOZILLA=1.596, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id AzgNB5Ff_t-x for ; Thu, 13 Jul 2017 15:59:39 +0000 (UTC) Received: from sonic306-23.consmr.mail.ne1.yahoo.com (sonic306-23.consmr.mail.ne1.yahoo.com [66.163.189.85]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 82F955F4A7 for ; Thu, 13 Jul 2017 15:59:39 +0000 (UTC) Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.ne1.yahoo.com with HTTP; Thu, 13 Jul 2017 15:59:33 +0000 Date: Thu, 13 Jul 2017 15:59:30 +0000 (UTC) From: Subroto Barua To: Message-ID: <1338506417.5323791.1499961570826@mail.yahoo.com> In-Reply-To: References: Subject: Re: nodetool removenode causing the schema out of sync MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_5323790_2135111208.1499961570821" X-Mailer: WebService/1.1.10014 YMailNorrin Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 archived-at: Thu, 13 Jul 2017 15:59:46 -0000 ------=_Part_5323790_2135111208.1499961570821 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable set streamthroughput higher than 200 on the source side and lower on the ta= rget node just curious, have you tried removenode force? On Thursday, July 13, 2017, 8:35:38 AM PDT, Jai Bheemsen Rao Dhanwada wrote: Thank you Sean, you mean setstreamthroughput to a lower value on the node where we are doin= g a "nodetool removenode "? On Thu, Jul 13, 2017 at 8:07 AM, Durity, Sean R wrote: Late to this party, but Jeff is talking about nodetool setstreamthroughput.= The default in most versions is 200 Mb/s (set in yaml file as stream_throu= ghput_outbound_ megabits_per_sec). This is outbound throttle only. So, if s= treams from multiple nodes are going to one, it can get inundated. =C2=A0 The nodetool command lets you change this on the fly (no bounce required), = but I don=E2=80=99t think it affects any current streaming from that node (= only future). You can use nodetool getstreamthroughput to see the current v= alue. =C2=A0 =C2=A0 Sean Durity =C2=A0 From: Jai Bheemsen Rao Dhanwada [mailto:jaibheemsen@gmail.com] Sent: Thursday, June 29, 2017 6:39 PM To: user@cassandra.apache.org Subject: Re: nodetool removenode causing the schema out of sync =C2=A0 Thanks Jeff, =C2=A0 Can you please suggest what value to tweak from the Cassandra side? =C2=A0 On Thu, Jun 29, 2017 at 2:53 PM, Jeff Jirsa wrote: On 2017-06-29 13:45 (-0700), Jai Bheemsen Rao Dhanwada wrote: > Hello Jeff, > > Sorry the Version I am using 2.1.16, my first email had typo. > When I say schema out of sync > > 1. nodetool descriebcluster shows Schema versions same for all nodes. Ok got it, this is what I was most concerned with. > 2. nodetool removenode, shows the node down messages in the logs > 3. nodetool describecluster during this 1-2 mins shows several nodes as > UNREACHABLE and recovers with in a minute or two. This is likely due to overhead of streaming - you're probably running prett= y close to your tipping point, and your streaming throughput creates enough= GC pressure on the destinations to make them flap a bit. If you use the st= reaming throughput throttle, you may be able to help mitigate that somewhat= (at the cost of speed). ------------------------------ ------------------------------ --------- To unsubscribe, e-mail: user-unsubscribe@cassandra. apache.org For additional commands, e-mail: user-help@cassandra.apache.org =C2=A0 The information in this Internet Email is confidential and may be legally p= rivileged. It is intended solely for the addressee. Access to this Email by= anyone else is unauthorized. If you are not the intended recipient, any di= sclosure, copying, distribution or any action taken or omitted to be taken = in reliance on it, is prohibited and may be unlawful. When addressed to our= clients any opinions or advice contained in this Email are subject to the = terms and conditions expressed in any applicable governing The Home Depot t= erms of business or client engagement letter. The Home Depot disclaims all = responsibility and liability for the accuracy and content of this attachmen= t and for any damages or losses arising from any inaccuracies, errors, viru= ses, e.g., worms, trojan horses, etc., or other items of a destructive natu= re, which may be contained in this attachment and shall not be liable for d= irect, indirect, consequential or special damages in connection with this e= -mail message or its attachment. ------=_Part_5323790_2135111208.1499961570821 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
set streamthroughput higher t= han 200 on the source side and lower on the target node

just curious, have you tried removenode force?


On Thursday, July 13, 2017, 8:35:38 AM PDT, = Jai Bheemsen Rao Dhanwada <jaibheemsen@gmail.com> wrote:

Thank you Sean,

you mean setstreamthroughput to a lower value on the node where w= e are doing a "nodetool removenode <hostid>"?=


On Thu, Jul 13, 2017 at 8:07 AM, Durity, Sean R = <SEAN_R_DURITY@homedepot.com> wrote:

Late = to this party, but Jeff is talking about nodetool setstreamthroughput. The = default in most versions is 200 Mb/s (set in yaml file as stream_throughput= _outbound_ megabits_per_sec). This is outbound throttle only. So, if streams from multiple nodes are goi= ng to one, it can get inundated.

=C2=A0

The n= odetool command lets you change this on the fly (no bounce required), but I= don=E2=80=99t think it affects any current streaming from that node (only = future). You can use nodetool getstreamthroughput to see the current value.

=C2=A0

=C2=A0

Sean = Durity

=C2=A0

Fr= om: Jai Bheemsen Rao Dhanwada = [mailto:jaibheemse= n@gmail.com]
Sent: Thursday, June 29, 2017 6:39 PM
To: user@cassandra.apache.org
Subject: Re: nodetool removenode causing the schema out of sync

=C2=A0

Thanks Jeff,

=C2=A0

Can you please suggest what value to tw= eak from the Cassandra side?

=C2=A0

On Thu, Jun 29, 2017 at 2:53 PM, Jeff J= irsa <jjirsa@apache.org= > wrote:



On 2017-06-29 13:45 (-0700), Jai Bheemsen Rao Dhanwada <jaibheemsen@gmail.com> wrot= e:
> Hello Jeff,
>
> Sorry the Version I am using 2.1.16, my first email had typo.
> When I say schema out of sync
>
> 1. nodetool descriebcluster shows Schema versions same for all nodes.<= br clear=3D"none">
Ok got it, this is what I was most concerned with.

> 2. nodetool removenode, shows the node down messages in the logs
> 3. nodetool describecluster during this 1-2 mins shows several nodes a= s
> UNREACHABLE and recovers with in a minute or two.

This is likely due to overhead of streaming - you're probably running p= retty close to your tipping point, and your streaming throughput creates en= ough GC pressure on the destinations to make them flap a bit. If you use th= e streaming throughput throttle, you may be able to help mitigate that somewhat (at the cost of speed).<= u>




------------------------------ ------------------------------ ---------
To unsubscribe, e-mail: user-unsubscribe@cassandra. apache.or= g
For additional commands, e-mail: user-help@cassandra.apache.org

=C2=A0



The information in this Internet Email is confidential and may be legally p= rivileged. It is intended solely for the addressee. Access to this Email by= anyone else is unauthorized. If you are not the intended recipient, any di= sclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibite= d and may be unlawful. When addressed to our clients any opinions or advice= contained in this Email are subject to the terms and conditions expressed = in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot d= isclaims all responsibility and liability for the accuracy and content of t= his attachment and for any damages or losses arising from any inaccuracies,= errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contain= ed in this attachment and shall not be liable for direct, indirect, consequ= ential or special damages in connection with this e-mail message or its att= achment.

=
------=_Part_5323790_2135111208.1499961570821--