cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: node replacement failed
Date Fri, 14 Sep 2018 10:16:43 GMT
Hello

Also i could delete system_traces which is empty anyway, but there is a
> system_auth and system_distributed keyspace too and they are not empty,
> Could i delete them safely too?


I would say no, not safely, as I am not sure about some of them, but maybe
this would work. Here is what I know: 'system_traces' can be deleted.
'system_auth'
should be ok to delete if you do not use auth (but I am not 100% sure)
but system_distributed
looks like an important keyspace, I am really uncertain about this one. It
was added in 'recent' Cassandra version and I am not too sure what's in
there, to be honest.

It seems quite a complex situation. A possible way out would be to do the 3
nodes replacement without streaming and then repair (making sure no client
talk to or get information from these nodes meanwhile), but I think
'replace' and 'auto_bootstrap: false' do not work well together.

Another idea is that 'nodetool removenode' on the 3 nodes, 1 by 1, should
also ensure consistency is preserved (if not forced). Then you could
recreate the 3 nodes of this rack. I am afraid this operation might fail as
well though, for the same reason of having ranges unavailable as well.

I am still thinking about it, but before going deeper, is this still an
issue for you at the moment?

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com


Le lun. 10 sept. 2018 à 13:43, onmstester onmstester <onmstester@zoho.com>
a écrit :

> Thanks Alain,
> First here it is more detail about my cluster:
>
>    - 10 racks + 3 nodes on each rack
>    - nodetool status: shows 27 nodes UN and 3 nodes all related to single
>    rack as DN
>    - version 3.11.2
>
> *Option 1: (Change schema and) use replace method (preferred method)*
> * Did you try to have the replace going, without any former repairs,
> ignoring the fact 'system_traces' might be inconsistent? You probably don't
> care about this table, so if Cassandra allows it with some of the nodes
> down, going this way is relatively safe probably. I really do not see what
> you could lose that matters in this table.
> * Another option, if the schema first change was accepted, is to make the
> second one, to drop this table. You can always rebuild it in case you need
> it I assume.
>
> I really love to let the replace going, but it stops with the error:
>
> java.lang.IllegalStateException: unable to find sufficient sources for
> streaming range in keyspace system_traces
>
>
> Also i could delete system_traces which is empty anyway, but there is a
> system_auth and system_distributed keyspace too and they are not empty,
> Could i delete them safely too?
> If i could just somehow skip streaming the system keyspaces from node
> replace phase, the option 1 would be great.
>
> P.S: Its clear to me that i should use at least RF=3 in production, but
> could not manage to acquire enough resources yet (i hope would be fixed in
> recent future)
>
> Again Thank you for your time
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
> ---- On Mon, 10 Sep 2018 16:20:10 +0430 *Alain RODRIGUEZ
> <arodrime@gmail.com <arodrime@gmail.com>>* wrote ----
>
> Hello,
>
> I am sorry it took us (the community) more than a day to answer to this
> rather critical situation. That being said, my recommendation at this point
> would be for you to make sure about the impacts of whatever you would try.
> Working on a broken cluster, as an emergency might lead you to a second
> mistake, possibly more destructive than the first one. It happened to me
> and around, for many clusters. Move forward even more carefuly in these
> situations as a global advice.
>
> Suddenly i lost all disks of cassandar-data on one of my racks
>
>
> With RF=2, I guess operations use LOCAL_ONE consistency, thus you should
> have all the data in the safe rack(s) with your configuration, you probably
> did not lose anything yet and have the service only using the nodes up,
> that got the right data.
>
>  tried to replace the nodes with same ip using this:
>
> https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html
>
>
> As a side note, I would recommend you to use 'replace_address_first_boot'
> instead of 'replace_address'. This does basically the same but will be
> ignored after the first bootstrap. A detail, but hey, it's there and
> somewhat safer, I would use this one.
>
> java.lang.IllegalStateException: unable to find sufficient sources for
> streaming range in keyspace system_traces
>
>
> By default, non-user keyspace use 'SimpleStrategy' and a small RF.
> Ideally, this should be changed in a production cluster, and you're having
> an example of why.
>
> Now when i altered the system_traces keyspace startegy to
> NetworkTopologyStrategy and RF=2
> but then running nodetool repair failed: Endpoint not alive /IP of dead
> node that i'm trying to replace.
>
>
> Changing the replication strategy you made the dead rack owner of part of
> the token ranges, thus repairs just can't work as there will always be one
> of the nodes involved down as the whole rack is down. Repair won't work,
> but you probably do not need it! 'system_traces' is a temporary / debug
> table. It's probably empty or with irrelevant data.
>
> Here are some thoughts:
>
> * It would be awesome at this point for us (and for you if you did not) to
> see the status of the cluster:
> ** 'nodetool status'
> ** *'nodetool describecluster' *--> This one will tell if the nodes agree
> on the schema (nodes up). I have seen schema changes with nodes down
> inducing some issues.
> *** *Cassandra version
> ** Number of racks (I assumer #racks >= 2 in this email)
>
> *Option 1: (Change schema and) use replace method (preferred method)*
> * Did you try to have the replace going, without any former repairs,
> ignoring the fact 'system_traces' might be inconsistent? You probably don't
> care about this table, so if Cassandra allows it with some of the nodes
> down, going this way is relatively safe probably. I really do not see what
> you could lose that matters in this table.
> * Another option, if the schema first change was accepted, is to make the
> second one, to drop this table. You can always rebuild it in case you need
> it I assume.
>
>
> *Option 2: Remove all the dead nodes *(try to avoid this option 2, if
> option 1 works, it is better).
>
> Please do not take an apply this like this. It's a thought on how you
> could get rid of the issue, yet it's rather brutal and risky and I did not
> consider it deeply and have no clue about your architecture and the
> context. Consider it carefully on your side.
>
> * You can also 'nodetool removenode' on each of the dead nodes. This will
> have nodes streaming around and the rack isolation guarantee will no longer
> be valid. It's hard to reason about what would happen to the data and in
> terms of streaming.
> * Alternatively, if you don't have enough space, you can even '*force*'
> the 'nodetool removenode'. See the documentation. Forcing it will prevent
> streaming and remove the node (token ranges handover, but not the data). If
> that does not work you can use the 'nodetool assassinate' command as well.
>
> When adding nodes back to the broken DC, the first nodes will take
> probably 100% of the ownership, which is often too much. You can consider
> adding back all the nodes with 'auto_bootstrap: false' before repairing
> them once they have their final token ownership, the same ways we do when
> building a new data center.
>
> This option is not really clean, and have some caveats that* you need to
> consider before starting* as there are token range movements and nodes
> available that do not have the data. Yet this should work. I imagine it
> would work nicely with RF=3 and QUORUM and with RF=2 (if you have 2+
> racks), I guess it should work as well but you will have to pick one of
> availability or consistency while repairing the data.
>
> *Be aware that read requests hitting these nodes will not find data!* Plus,
> you are using an *RF=2*. Thus using consistency of 2+ (TWO, QUORUM, ALL),
> for at least one of reads or writes is needed to preserve consistency while
> re-adding the nodes in this case. Otherwise, reads will not detect the
> mismatch with certainty and might show inconsistent data the time for the
> nodes to be repaired.
>
> I must say, that I really prefer odd values for the RF, starting with
> RF=3. Using RF=2 you will have to pick. Consistency or Availability. With a
> consistency of ONE everywhere, the service is available, no single point of
> failure. using anything bigger than this, for writes or read, brings
> consistency but it creates single points of failures (actually any node
> becomes a point of failure). RF=3 and QUORUM for both write and reads
> take the best of the 2 worlds somehow. The tradeoff with RF=3 and quorum
> reads is the latency increase and the resource usage.
>
> Maybe is there a better approach, I am not too sure, but I think I would
> try option 1 first in any case. It's less destructive, less risky, no token
> range movements, no empty nodes available. I am not sure about limitation
> you might face though and that's why I suggest a second option for you to
> consider if the first is not actionable.
>
> Let us know how it goes,
> C*heers,
> -----------------------
> Alain Rodriguez - @arodream - alain@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> Le lun. 10 sept. 2018 à 09:09, onmstester onmstester <onmstester@zoho.com>
> a écrit :
>
>
> Any idea?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
> ---- On Sun, 09 Sep 2018 11:23:17 +0430 *onmstester onmstester
> <onmstester@zoho.com <onmstester@zoho.com>>* wrote ----
>
>
> Hi,
>
> Cluster Spec:
> 30 nodes
> RF = 2
> NetworkTopologyStrategy
> GossipingPropertyFileSnitch + rack aware
>
> Suddenly i lost all disks of cassandar-data on one of my racks, after
> replacing the disks, tried to replace the nodes with same ip using this:
>
> https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html
>
> starting the to-be-replace-node fails with:
> java.lang.IllegalStateException: unable to find sufficient sources for
> streaming range in keyspace system_traces
>
> the problem is that i did not changed default replication config for
> System keyspaces, but Now when i altered the system_traces keyspace
> startegy to NetworkTopologyStrategy and RF=2
> but then running nodetool repair failed: Endpoint not alive /IP of dead
> node that i'm trying to replace.
>
> What should i do now?
> Can i just remove previous nodes, change dead nodes IPs and re-join them
> to cluster?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>
>
>

Mime
View raw message