incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Sargisson <edward.sargis...@globalrelay.net>
Subject Re: How to replace a dead *seed* node while keeping quorum
Date Wed, 12 Sep 2012 23:42:57 GMT
I'm reposting my colleague's reply to Rob to the list (with James' 
permission) in case others are interested.

I'll add to James' post below to say I don't believe we saw the message 
that that slice of code would have printed.

"

Hey Rob,

Ed's AWOL right now and I'm not onu@c.a.o, but I can tell you that when
I removed the downed seed node from its own list of seed nodes in
cassandra.yaml that it didn't join the existing ring nor did it get any
schemas or data from the existing ring; it felt like timeouts were
happening. (IANA Cassandra wizard, so excuse my terminology impedance.)

Changing the machine's hostname and giving it a new IP, it behaved as
expected; joining the ring, syncing both schema and associated data.

Downed node is 1.1.4, the rest of the ring is 1.1.2.

I'm in a situation where I can revert the IP/hostname change and retry
the scenario as needed if you've got any ideas.

HTH,

    JAmes"


Cheers,
Edward

On 12-09-12 03:53 PM, Rob Coli wrote:
> On Tue, Sep 11, 2012 at 4:21 PM, Edward Sargisson
> <edward.sargisson@globalrelay.net> wrote:
>> If the downed node is a seed node then neither of the replace a dead node
>> procedures work (-Dcassandra.replace_token and taking initial_token-1). The
>> ring remains split.
>> [...]
>> In other words, if the host name is on the seeds list then it appears that
>> the rest of the ring refuses to bootstrap it.
> Close, but not exactly...
>
> "./src/java/org/apache/cassandra/service/StorageService.java" line 559 of 3090
> "
> if (DatabaseDescriptor.isAutoBootstrap()
>                  &&
> DatabaseDescriptor.getSeeds().contains(FBUtilities.getBroadcastAddress())
>                  && !SystemTable.isBootstrapped())
>              logger_.info("This node will not auto bootstrap because it
> is configured to be a seed node.");
> "
>
> getSeeds asks your seed provider for a list of seeds. If you are using
> the SimpleSeedProvider, this basically turns the list from "seeds" in
> cassandra.yaml on the local node into a list of hosts.
>
> So it isn't that the other nodes have this node in their seed list..
> it's that the node you are replacing has itself in its own seed list,
> and shouldn't. I understand that it can be tricky in conf management
> tools to make seed nodes' seed lists not contain themselves, but I
> believe it is currently necessary in this case.
>
> FWIW, it's unclear to me (and Aaron Morton, whose curiousity was
> apparently equally piqued and is looking into it further..) why
> exactly seed nodes shouldn't bootstrap. It's possible that they only
> shouldn't bootstrap without being in "hibernate" mode, and that the
> code just hasn't been re-written post replace_token/hibernate to say
> that it's ok for seed nodes to bootstrap as long as they hibernate...
>
> =Rob
>

-- 

Edward Sargisson

senior java developer
Global Relay

edward.sargisson@globalrelay.net <mailto:edward.sargisson@globalrelay.net>


*866.484.6630*
New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore 
(+65.3158.1301)

Global Relay Archive supports email, instant messaging, BlackBerry, 
Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, 
Facebook and more.


Ask about *Global Relay Message* 
<http://www.globalrelay.com/services/message>*--- *The Future of 
Collaboration in the Financial Services World

*
*All email sent to or from this address will be retained by Global 
Relay's email archiving system. This message is intended only for the 
use of the individual or entity to which it is addressed, and may 
contain information that is privileged, confidential, and exempt from 
disclosure under applicable law.  Global Relay will not be liable for 
any compliance or technical information provided herein. All trademarks 
are the property of their respective owners.


Mime
View raw message