cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peddi, Praveen" <pe...@amazon.com>
Subject Re: Removing Node causes bunch of HostUnavailableException
Date Thu, 03 Mar 2016 21:40:07 GMT
Hi Jack,
Which node(s) were getting the HostNotAvailable errors - all nodes for every query, or just
a small portion of the nodes on some queries?
Not all read/writes are failing with Unavalable or Timeout exception. Writes failures were
around 10% of total calls. Reads were little worse (as worse as 35% of total calls).


It may take some time for the gossip state to propagate; maybe some of it is corrupted or
needs a full refresh.

Were any of the seed nodes in the collection of nodes that were removed? How many seed nodes
does each node typically have?
We currently use all hosts as seed hosts which I know is a very bad idea and we are going
to fix that soon. The reason we use all hosts as seed hosts is because these hosts can get
recycled for many reasons and we didn’t want to hard code the host names so we programmatically
get host names (we wrote our own seed host provider). Could that be the reason for these failures?
If a dead node is in the seed nodes list and we try to remove that node, could that lead to
blip of failures. The failures don’t last for more than few minutes.


-- Jack Krupansky

On Thu, Mar 3, 2016 at 4:16 PM, Peddi, Praveen <peddi@amazon.com<mailto:peddi@amazon.com>>
wrote:
Thanks Alain for quick and detailed response. My answers inline. One thing I want to clarify
is, the nodes got recycled due to some automatic health check failure. This means old nodes
are dead and new nodes got added w/o our intervention. So replacing nodes would not work for
us since the new nodes were already added.


We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there
were no errors when the nodes were down as expected (because we use QUORUM)

Do you use at leat 3 distinct AZ ? If so, you should indeed be fine regarding data integrity.
Also repair should then work for you. If you have less than 3 AZ, then you are in troubles...
Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when 8 nodes were recycled,
there were absolutely no outage on Cassandra (other two nodes wtill satisfy quorum consistency)

About the unreachable errors, I believe it can be due to the overload due to the missing nodes.
Pressure on the remaining node might be too strong.
It is certainly possible but we have beefed up cluster with <3% CPU, hardly any network
I/o and disk usage. We have 162 nodes in the cluster and each node doesn’t have more than
80 to 100MB of data.


However, As soon as I started removing nodes one by one, every time time we see lot of timeout
and unavailable exceptions which doesn’t make any sense because I am just removing a node
that doesn’t even exist.

This probably added even more load, if you are using vnodes, all the remaining nodes probably
started streaming data to each other node at the speed of "nodetool getstreamthroughput".
AWS network isn't that good, and is probably saturated. Also have you the phi_convict_threshold
configured to a high value at least 10 or 12 ? This would avoid nodes to be marked down that
often.
We are using c3.2xlarge which has good network throughput (1GB/sec I think). We are using
default value which is 200MB/sec in 2.0.9. We will play with it in future and see if this
could make any difference but as I mentioned the data size on each node is not huge.
Regarding phi_convict_threshold, our Cassandra is not bringing itself down. There was a bug
in health check from one of our internal tool and that tool is recycling the nodes. Nothing
to do with Cassandra health. Again we will keep an eye on it in future.


What does "nodetool tpstats" outputs ?
Nodetool tpstats on which node? Any node?


Also you might try to monitor resources and see what happens (my guess is you should focus
at iowait, disk usage and network, have an eye at cpu too).
We did monitor cpu, disk and network and they are all very low.


A quick fix would probably be to hardly throttle the network on all the nodes and see if it
helps:

nodetool setstreamthroughput 2
We will play with this config. 2.0.9 defaults to 200MB/sec which I think is too high.


If this work, you could incrementally increase it and monitor, find the good tuning and put
it the cassandra.yaml.

I opened a ticket a while ago about that issue: https://issues.apache.org/jira/browse/CASSANDRA-9509
I voted for this issue. Lets see if it gets picked up :).


I hope this will help you to go back to a healthy state allowing you a fast upgrade ;-).

C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com<mailto:alain@thelastpickle.com>
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-03-02 22:17 GMT+01:00 Peddi, Praveen <peddi@amazon.com<mailto:peddi@amazon.com>>:
Hi Robert,
Thanks for your response.

Replication factor is 3.

We are in the process of upgrading to 2.2.4. We have had too many performance issues with
later versions of Cassandra (I have asked asked for help related to that in the forum). We
are close to getting to similar performance now and hopefully upgrade in next few weeks. Lot
of testing to do :(.

We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there
were no errors when the nodes were down as expected (because we use QUORUM). However, As soon
as I started removing nodes one by one, every time time we see lot of timeout and unavailable
exceptions which doesn’t make any sense because I am just removing a node that doesn’t
even exist.








From: Robert Coli <rcoli@eventbrite.com<mailto:rcoli@eventbrite.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Wednesday, March 2, 2016 at 2:52 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Removing Node causes bunch of HostUnavailableException

On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <peddi@amazon.com<mailto:peddi@amazon.com>>
wrote:
We have few dead nodes in the cluster (Amazon ASG removed those thinking there is an issue
with health). Now we are trying to remove those dead nodes from the cluster so that other
nodes can take over. As soon as I execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions
both on reads and writes. What I am not able to understand is, these are deadnodes and don’t
even physically exists. Why would removenode command cause any outage of nodes in Cassandra
when we had no errors whatsoever before removing them. I could not really find a jira ticket
for this.

What is your replication factor?

Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.

Also, removing multiple nodes with removenode means your consistency is pretty hosed. Repair
ASAP, but there are potential cases where repair won't help.

=Rob


=Rob




Mime
View raw message