cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edmond Lau <edm...@ooyala.com>
Subject Re: on bootstrapping a node
Date Wed, 28 Oct 2009 19:15:12 GMT
On Tue, Oct 27, 2009 at 10:02 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
> On Tue, Oct 27, 2009 at 10:10 PM, Edmond Lau <edmond@ooyala.com> wrote:
>> I'd like to improve my mental model of how Cassandra bootstrapping
>> works.  My understanding is that bootstrapping is just an extra step
>> during a node's startup where the node copies data from neighboring
>> nodes that, according to its token, it should own; afterwards, the
>> node behaves like any other node.
>>
>> If that's correct, I have a few follow-up questions:
>>
>> - At what point does the new node get inserted into the hash ring so
>> that reads/writes for keys get directed to it?
>
> It starts accepting reads when bootstrap is complete, but during
> bootstrap it will also be forwarded writes so it stays up-to-date.

Got it - good to know.

>
>> - What are the semantics of bootstrapping a node that's been in the
>> cluster before and already has some data that's possibly outdated?
>> Should this work?  This might be useful if a node's been out of
>> commission for sufficiently long period of time.
>
> Cluster membership is intended to be relatively stable.  When removing
> nodes is supported (see
> https://issues.apache.org/jira/browse/CASSANDRA-435), it won't be
> worth special-casing the case of "new node, that has some data locally
> that is arbitrarily out of date."  So you will want to wipe its data
> directories and re-bootstrap it.

Sounds reasonable.  Until CASSANDRA-435 is complete, there's no way
currently to take down a node and have it be removed from the list of
nodes that's responsible for the data in its token range, correct?
All other nodes will just assume that it's temporarily unavailable?

>
>> - If we pick a poor set of initial tokens, would it be sensible to
>> modify the tokens on existing nodes and then restart them with
>> bootstrapping in order to rebalance?
>
> Not until https://issues.apache.org/jira/browse/CASSANDRA-193 is done
> so you can fix the damage you've done to your replicas.

Assume that we had the ability to permanently remove a node.  Would
modifying the token on an existing node and restarting it with
bootstrapping somehow be incorrect, or merely not performant b/c we'll
be performing lazy repair on most reads until the node is up to date?

>
>> I've also noticed that I can get my cassandra cluster into a weird
>> state via bootstrapping, where it stops accepting reads/writes.  I'm
>> on Cassandra 0.4.1.  A simple repro case is to start all 3 nodes of a
>> 3 node cluster (replication factor of 2) using bootstrapping.  Getting
>> a key that I've inserted then leads to an IndexOutOfBoundsException.
>> Another IndexOutOfBoundsException was thrown later while flushing.
>
> Starting all nodes in bootstrap mode is not a supported operation.
> Don't do that.  (Bootstrap is much more automatic in trunk, and fixes
> this specific problem as well as others.)

The unsupported operation is that all nodes in the cluster are
simultaneously in bootstrap mode, and not that all nodes in the
cluster were bootstrapped at some point in time, right?  A reasonable
scenario that could cause the second situation would be if I wanted to
migrate my cluster to a completely new set of machines.  I would then
bootstrap all the new nodes in the new cluster, and then decommission
my old nodes one by one (assuming
https://issues.apache.org/jira/browse/CASSANDRA-435 was done).  After
the migration, all my nodes would've been bootstrapped.

Thanks for the info,
Edmond

>
> -Jonathan
>

Mime
View raw message