Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5311E19CCE for ; Tue, 15 Mar 2016 17:55:48 +0000 (UTC) Received: (qmail 49384 invoked by uid 500); 15 Mar 2016 17:55:46 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 49335 invoked by uid 500); 15 Mar 2016 17:55:46 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 49325 invoked by uid 99); 15 Mar 2016 17:55:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2016 17:55:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id ADCC71A00D2 for ; Tue, 15 Mar 2016 17:55:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -10.95 X-Spam-Level: X-Spam-Status: No, score=-10.95 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.329, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=amazon.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id KjBi8q2w90YS for ; Tue, 15 Mar 2016 17:55:42 +0000 (UTC) Received: from smtp-fw-33001.amazon.com (smtp-fw-33001.amazon.com [207.171.189.228]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id B2F695F19D for ; Tue, 15 Mar 2016 17:55:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1458064540; x=1489600540; h=from:to:subject:date:message-id:references:in-reply-to: mime-version; bh=66Jn/395+sIh5ttup1h3pNgHdrd2DlXSUCtvr7HxR5g=; b=UeVPO51NUDG+LuMaKfssEdV+ncNdIh4rOKg7xk+nw2pbMx2fMpugXvrc zWPK1FjXQILgcJZ16SZFvsr91RbBF4+vUFLK6i7gtJq6Hys7AW84qKvva NuBzgybr0vuzphpr+xSCInIhISlg5KqkMQ6cshBtpoujk2THZsAoCxrSC Y=; X-IronPort-AV: E=Sophos;i="5.24,340,1454976000"; d="scan'208,217";a="460661540" Received: from sea19-co-svc-lb5-vlan3.sea.amazon.com (HELO email-inbound-relay-60016.pdx1.amazon.com) ([10.47.22.166]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 15 Mar 2016 17:55:38 +0000 Received: from ex10-hub-7001.ant.amazon.com (pdx1-ws-svc-lb16-vlan2.amazon.com [10.239.138.210]) by email-inbound-relay-60016.pdx1.amazon.com (8.14.7/8.14.7) with ESMTP id u2FHtYlh027755 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL) for ; Tue, 15 Mar 2016 17:55:37 GMT Received: from EX13D02UEB004.ant.amazon.com (10.43.60.108) by ex10-hub-7001.ant.amazon.com (10.43.103.49) with Microsoft SMTP Server (TLS) id 14.3.181.6; Tue, 15 Mar 2016 10:55:16 -0700 Received: from EX13D02UEB004.ant.amazon.com (10.43.60.108) by EX13D02UEB004.ant.amazon.com (10.43.60.108) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Tue, 15 Mar 2016 17:55:16 +0000 Received: from EX13D02UEB004.ant.amazon.com ([10.43.60.108]) by EX13D02UEB004.ant.amazon.com ([10.43.60.108]) with mapi id 15.00.1104.000; Tue, 15 Mar 2016 17:55:16 +0000 From: "Peddi, Praveen" To: "user@cassandra.apache.org" Subject: Re: Removing Node causes bunch of HostUnavailableException Thread-Topic: Removing Node causes bunch of HostUnavailableException Thread-Index: AQHRdJ4Ud3J0GGm5J0uTFwkVPZDx8p9GkPcA///D0QCAAG/KAIABIiiAgABXEoD//6+mAIAA1fqAgACJ9gCACTkOAIAIJbAA Date: Tue, 15 Mar 2016 17:55:15 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.43.60.113] Content-Type: multipart/alternative; boundary="_000_D30DC3053CFCDpeddiamazoncom_" MIME-Version: 1.0 --_000_D30DC3053CFCDpeddiamazoncom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Alain, Sorry I completely missed your email until my colleague pointed it out. >From the testing we have done so far, We still have this issue when removin= g nodes on 2.0.9 but not on 2.2.4. We will be upgrading to 2.2.4 pretty soo= n so I am not too worried about errors in 2.0.9. Since we haven=92t changed= any code on our side between 2.0.9 and 2.2.4, I am guessing this is probab= ly a bug in 2.0.9 and since 2.0.9 is not supported anymore, I didn=92t crea= te a jira ticket for it. Thanks for following up though. Praveen From: Alain RODRIGUEZ > Reply-To: "user@cassandra.apache.org" > Date: Thursday, March 10, 2016 at 5:30 AM To: "user@cassandra.apache.org" > Subject: Re: Removing Node causes bunch of HostUnavailableException Hi Praveen, how is this going ? I have been out for a while, did you manage to remove the nodes ? Do you ne= ed more help ? If so, I could use a status update and more information abou= t the remaining issues. C*heers, ----------------------- Alain Rodriguez - alain@thelastpickle.com France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2016-03-04 19:39 GMT+01:00 Peddi, Praveen >: Hi Jack, My answers below=85 What is the exact exception you are getting and where do you get it? Is it = UnavailableException or NoHostAvailableException and does it occur on the c= lient, using the Java driver? We saw different types of exceptions. One I could quickly grep are: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeou= t during write query at consistency SERIAL (2 replica were required but onl= y 1 acknowledged the write) com.datastax.driver.core.exceptions.UnavailableException: Not enough replic= a available for query at consistency QUORUM (2 required but only 1 alive) QueryTimeoutException What is your LoadBalancingPolicy? new TokenAwarePolicy(new RoundRobinPolicy())) What consistency level is the client using? QUORUM for reads. For writes some APIs use SERIAL and some use QUORUM depen= dending if we want to do optimistic locking, What retry policy is the client using? Default Retry Policy When you say that the failures don't last for more than a few minutes, you = mean from the moment you perform the nodetool removenode? And is operation = completely normal after those few minutes? That is correct. All operations recover from failures after few minutes. -- Jack Krupansky On Thu, Mar 3, 2016 at 4:40 PM, Peddi, Praveen > wrote: Hi Jack, Which node(s) were getting the HostNotAvailable errors - all nodes for ever= y query, or just a small portion of the nodes on some queries? Not all read/writes are failing with Unavalable or Timeout exception. Write= s failures were around 10% of total calls. Reads were little worse (as wors= e as 35% of total calls). It may take some time for the gossip state to propagate; maybe some of it i= s corrupted or needs a full refresh. Were any of the seed nodes in the collection of nodes that were removed? Ho= w many seed nodes does each node typically have? We currently use all hosts as seed hosts which I know is a very bad idea an= d we are going to fix that soon. The reason we use all hosts as seed hosts = is because these hosts can get recycled for many reasons and we didn=92t wa= nt to hard code the host names so we programmatically get host names (we wr= ote our own seed host provider). Could that be the reason for these failure= s? If a dead node is in the seed nodes list and we try to remove that node,= could that lead to blip of failures. The failures don=92t last for more th= an few minutes. -- Jack Krupansky On Thu, Mar 3, 2016 at 4:16 PM, Peddi, Praveen > wrote: Thanks Alain for quick and detailed response. My answers inline. One thing = I want to clarify is, the nodes got recycled due to some automatic health c= heck failure. This means old nodes are dead and new nodes got added w/o our= intervention. So replacing nodes would not work for us since the new nodes= were already added. We are not removing multiple nodes at the same time. All dead nodes are fro= m same AZ so there were no errors when the nodes were down as expected (bec= ause we use QUORUM) Do you use at leat 3 distinct AZ ? If so, you should indeed be fine regardi= ng data integrity. Also repair should then work for you. If you have less t= han 3 AZ, then you are in troubles... Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when 8 no= des were recycled, there were absolutely no outage on Cassandra (other two = nodes wtill satisfy quorum consistency) About the unreachable errors, I believe it can be due to the overload due t= o the missing nodes. Pressure on the remaining node might be too strong. It is certainly possible but we have beefed up cluster with <3% CPU, hardly= any network I/o and disk usage. We have 162 nodes in the cluster and each = node doesn=92t have more than 80 to 100MB of data. However, As soon as I started removing nodes one by one, every time time we= see lot of timeout and unavailable exceptions which doesn=92t make any sen= se because I am just removing a node that doesn=92t even exist. This probably added even more load, if you are using vnodes, all the remain= ing nodes probably started streaming data to each other node at the speed o= f "nodetool getstreamthroughput". AWS network isn't that good, and is proba= bly saturated. Also have you the phi_convict_threshold configured to a high= value at least 10 or 12 ? This would avoid nodes to be marked down that of= ten. We are using c3.2xlarge which has good network throughput (1GB/sec I think)= . We are using default value which is 200MB/sec in 2.0.9. We will play with= it in future and see if this could make any difference but as I mentioned = the data size on each node is not huge. Regarding phi_convict_threshold, our Cassandra is not bringing itself down.= There was a bug in health check from one of our internal tool and that too= l is recycling the nodes. Nothing to do with Cassandra health. Again we wil= l keep an eye on it in future. What does "nodetool tpstats" outputs ? Nodetool tpstats on which node? Any node? Also you might try to monitor resources and see what happens (my guess is y= ou should focus at iowait, disk usage and network, have an eye at cpu too). We did monitor cpu, disk and network and they are all very low. A quick fix would probably be to hardly throttle the network on all the nod= es and see if it helps: nodetool setstreamthroughput 2 We will play with this config. 2.0.9 defaults to 200MB/sec which I think is= too high. If this work, you could incrementally increase it and monitor, find the goo= d tuning and put it the cassandra.yaml. I opened a ticket a while ago about that issue: https://issues.apache.org/j= ira/browse/CASSANDRA-9509 I voted for this issue. Lets see if it gets picked up :). I hope this will help you to go back to a healthy state allowing you a fast= upgrade ;-). C*heers, ----------------------- Alain Rodriguez - alain@thelastpickle.com France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2016-03-02 22:17 GMT+01:00 Peddi, Praveen >: Hi Robert, Thanks for your response. Replication factor is 3. We are in the process of upgrading to 2.2.4. We have had too many performan= ce issues with later versions of Cassandra (I have asked asked for help rel= ated to that in the forum). We are close to getting to similar performance = now and hopefully upgrade in next few weeks. Lot of testing to do :(. We are not removing multiple nodes at the same time. All dead nodes are fro= m same AZ so there were no errors when the nodes were down as expected (bec= ause we use QUORUM). However, As soon as I started removing nodes one by on= e, every time time we see lot of timeout and unavailable exceptions which d= oesn=92t make any sense because I am just removing a node that doesn=92t ev= en exist. From: Robert Coli > Reply-To: "user@cassandra.apache.org" > Date: Wednesday, March 2, 2016 at 2:52 PM To: "user@cassandra.apache.org" > Subject: Re: Removing Node causes bunch of HostUnavailableException On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen > wrote: We have few dead nodes in the cluster (Amazon ASG removed those thinking th= ere is an issue with health). Now we are trying to remove those dead nodes = from the cluster so that other nodes can take over. As soon as I execute no= detool removenode , we see lots of HostUnavailableExceptions both on re= ads and writes. What I am not able to understand is, these are deadnodes an= d don=92t even physically exists. Why would removenode command cause any ou= tage of nodes in Cassandra when we had no errors whatsoever before removing= them. I could not really find a jira ticket for this. What is your replication factor? Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP. Also, removing multiple nodes with removenode means your consistency is pre= tty hosed. Repair ASAP, but there are potential cases where repair won't he= lp. =3DRob =3DRob --_000_D30DC3053CFCDpeddiamazoncom_ Content-Type: text/html; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable
Hi Alain,
Sorry I completely missed your email until my colleague pointed it out= .

From the testing we have done so far, We still have this issue when re= moving nodes on 2.0.9 but not on 2.2.4. We will be upgrading to 2.2.4 prett= y soon so I am not too worried about errors in 2.0.9. Since we haven=92t ch= anged any code on our side between 2.0.9 and 2.2.4, I am guessing this is probably a bug in 2.0.9 and since 2= .0.9 is not supported anymore, I didn=92t create a jira ticket for it.

Thanks for following up though.

Praveen


From: Alain RODRIGUEZ <arodrime@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Thursday, March 10, 2016 at 5= :30 AM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Removing Node causes b= unch of HostUnavailableException

Hi Praveen, how is this going ?

I have been out for a while, did you manage to remove the nodes ? Do y= ou need more help ? If so, I could use a status update and more information= about the remaining issues.

C*heers,
-----------------------
Alain Rodriguez - alain@the= lastpickle.com
France

The Last Pickle - Apache Cassandra Consulting

2016-03-04 19:39 GMT+01:00 Peddi, Praveen <peddi@amazon.com<= /a>>:
Hi Jack,
My answers below=85

What is the exact exception you are getting and where do y= ou get it? Is it UnavailableException or NoHostAvailableException and = does it occur on the client, using the Java driver?
We saw different types of exceptions. One I could quickly grep are:
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra t= imeout during write query at consistency SERIAL (2 replica were required bu= t only 1 acknowledged the write)
com.datastax.driver.core.exceptions.UnavailableException: Not enough r= eplica available for query at consistency QUORUM (2 required but only 1 ali= ve)
QueryTimeoutException



What is your LoadBalancingPolicy?
 
new TokenAwarePolicy(new<=
/span> RoundRobinPolicy()))

What consistency level is the client using?
QUORUM for reads. For writes some APIs use SERIAL and some use QUORUM = dependending if we want to do optimistic locking,

What retry policy is the client using?
Default Retry Policy


When you say that the failures don't last for more than a few minutes,= you mean from the moment you perform the nodetool removenode? And is opera= tion completely normal after those few minutes?
That is correct. All operations recover from failures after few minute= s.



-- Jack Krupansky

On Thu, Mar 3, 2016 at 4:40 PM, Peddi, Praveen <= span dir=3D"ltr"> <peddi@amazon.com<= /a>> wrote:
Hi Jack,
Which node(s) were getting the HostNotAvailable errors - a= ll nodes for every query, or just a small portion of the nodes on some quer= ies?
Not all read/writes are failing with Unavalable or Timeout exception. = Writes failures were around 10% of total calls. Reads were little worse (as= worse as 35% of total calls).


It may take some time for the gossip state to propagate; maybe some of= it is corrupted or needs a full refresh.

Were any of the seed nodes in the collection of nodes that were remove= d? How many seed nodes does each node typically have?
We currently use all hosts as seed hosts which I know is a very bad id= ea and we are going to fix that soon. The reason we use all hosts as seed h= osts is because these hosts can get recycled for many reasons and we didn= =92t want to hard code the host names so we programmatically get host names (we wrote our own seed host provider= ). Could that be the reason for these failures? If a dead node is in the se= ed nodes list and we try to remove that node, could that lead to blip of fa= ilures. The failures don=92t last for more than few minutes.


-- Jack Krupansky

On Thu, Mar 3, 2016 at 4:16 PM, Peddi, Praveen <= span dir=3D"ltr"> <peddi@amazon.com<= /a>> wrote:
Thanks Alain for quick and detailed response. My answers inline. One t= hing I want to clarify is, the nodes got recycled due to some automatic hea= lth check failure. This means old nodes are dead and new nodes got added w/= o our intervention. So replacing nodes would not work for us since the new nodes were already added.

 
We are not removing multiple nodes at the same time. All dead = nodes are from same AZ so there were no errors when the nodes were down as = expected (because we use QUORUM)

Do you use at leat 3 distinct AZ ? If so, you should indeed be fine re= garding data integrity. Also repair should then work for you. If you have l= ess than 3 AZ, then you are in troubles...
Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when= 8 nodes were recycled, there were absolutely no outage on Cassandra (other= two nodes wtill satisfy quorum consistency)

About the unreachable errors, I believe it can be due to the overload = due to the missing nodes. Pressure on the remaining node might be too stron= g.
It is certainly possible but we have beefed up cluster with <3% CPU= , hardly any network I/o and disk usage. We have 162 nodes in the cluster a= nd each node doesn=92t have more than 80 to 100MB of data.


However, As soon as I started removing nodes one by one, every time time= we see lot of timeout and unavailable exceptions which doesn=92t make any = sense because I am just removing a node that doesn=92t even exist.

This probably added even more load, if you are using vnodes, all the r= emaining nodes probably started streaming data to each other node at the sp= eed of "nodetool getstreamthroughput". AWS network isn't that goo= d, and is probably saturated. Also have you the phi_convict_threshold configured to a high value at least 10 or 12 ? T= his would avoid nodes to be marked down that often.
We are using c3.2xlarge which has good network throughput (1GB/sec I t= hink). We are using default value which is 200MB/sec in 2.0.9. We will play= with it in future and see if this could make any difference but as I menti= oned the data size on each node is not huge.
Regarding phi_convict_threshold, our Cassandra is not bringing itself = down. There was a bug in health check from one of our internal tool and tha= t tool is recycling the nodes. Nothing to do with Cassandra health. Again w= e will keep an eye on it in future.


What does "nodetool tpstats" outputs ?
Nodetool tpstats on which node? Any node?


Also you might try to monitor resources and see what happens (my guess= is you should focus at iowait, disk usage and network, have an eye at cpu = too).
We did monitor cpu, disk and network and they are all very low.


A quick fix would probably be to hardly throttle the network on all th= e nodes and see if it helps:

nodetool setstreamthroughput 2
We will play with this config. 2.0.9 defaults to 200MB/sec which I thi= nk is too high.

I voted for this issue. Lets see if it gets picked up :).


2016-03-02 22:17 GMT+01:00 Peddi, Praveen <peddi@amazon.com<= /a>>:
Hi Robert,
Thanks for your response.

Replication factor is 3.

We are in the process of upgrading to 2.2.4. We have had too many perf= ormance issues with later versions of Cassandra (I have asked asked for hel= p related to that in the forum). We are close to getting to similar perform= ance now and hopefully upgrade in next few weeks. Lot of testing to do :(.

We are not removing multiple nodes at the same time. All dead nodes ar= e from same AZ so there were no errors when the nodes were down as expected= (because we use QUORUM). However, As soon as I started removing nodes one = by one, every time time we see lot of timeout and unavailable exceptions which doesn=92t make any sense becau= se I am just removing a node that doesn=92t even exist.




On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <= span dir=3D"ltr"> <peddi@amazon.com<= /a>> wrote:
We have few dead nodes in the cluster (Amazon ASG removed those thinki= ng there is an issue with health). Now we are trying to remove those dead n= odes from the cluster so that other nodes can take over. As soon as I execu= te nodetool removenode <ID>, we see lots of HostUnavailableExceptions both on reads and writes. What I am = not able to understand is, these are deadnodes and don=92t even physically = exists. Why would removenode command cause any outage of nodes in Cassandra= when we had no errors whatsoever before removing them. I could not really find a jira ticket for this.

What is your replication factor?

Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP= .

Also, removing multiple nodes with removenode means your consistency i= s pretty hosed. Repair ASAP, but there are potential cases where repair won= 't help.

=3DRob


=3DRob
 




 --_000_D30DC3053CFCDpeddiamazoncom_--