incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip Stanhope <pstanh...@wimba.com>
Subject Re: Seeds, autobootstrap nodes, and replication factor
Date Fri, 04 Jun 2010 18:04:09 GMT
Thanks on the correction about Keyspace versus ColumnFamily ... I knew that just mis-typed.

I guess it should be stated (to be obvious) ... that when you are auto bootstrapping a node
... the seed better be alive. The scenario I'm dealing with is that it might not be (reasons
for that are tangential). 

I am contemplating a situation where there may be 2N servers ... but only N online at any
one time. But, for operational purposes, N+n (where n is 1 or 2), N may be occasionally greater
than R. 

This gets to the hinted hand-off question ... if R=8 and N=8 ... and all was fine for awhile
... and then N8(S8)  (node 8, server 8) goes down ... N8(S9) replaces it ... N8(S9) will take
the hit to obtain all that it never had before. Then ... at some subsequent time, N9(S8) comes
back to life ... will it take over its former role and the R is now 9 even though the storage-conf
had set it to 8 for a particular keyspace?

I'm asking these questions ... because they've been asked of me. I've been working with Cassandra
for 3+ months now and this level of the key management is something that I struggle to get
my head around.

What do you mean by "token automatic assignment may not do what you want"? If I specify R=N
... then what I want is all data to replicated to all nodes. What does a power of 2 have to
do with this? Are there undocumented recommendations about cluster size to ensure that you
can survive any one (or two) nodes failing?

Thanks in advance.

-phil

On Jun 4, 2010, at 1:46 PM, Benjamin Black wrote:

> On Fri, Jun 4, 2010 at 10:36 AM, Philip Stanhope <pstanhope@wimba.com> wrote:
>> 
>> Here's the scenario: would like R = N where N is the number of nodes. Let's say 8.
>> 
>> 1. Create first node, modify storage-conf.xml and change the <Seed/> to be
the ip of the node. Change replication factor to 8 for CF of interest. Start the puppy up.
>> 
> 
> RF is per Keyspace, not per CF.
> 
>> 2. Create 2nd node, modify storage-confg.xml and change <AutoBootstrap/> to
true and let it know the first seed. Ensure replication factor is 8 for the CF of interest.
Start the puppy up.
>> 
> 
> If you do it this way be aware token automatic assignment may not do
> what you want.  It _probably_ will, since 8 is a power of 2, but be
> aware.
> 
>> 3. Create 3rd node.
>> 
>> Q1: Should the node1 and node2 be listed as seeds? Or only node1?
>> 
> 
> Doesn't matter.  Seeds are only used as a discovery mechanism.  One is
> sufficient.
> 
>> 4. Create 4th node. Same question as before.
>> 
>> Q2: Same question as before ... should the list of seeds grow as the cluster grows?
Alternative phrasing ... what is the relationship between Seed and AutoBootstrap, i.e. can
a Seed node in fact be a node that was autobootstrapped? Is this considered best practice?
>> 
> 
> Once a node is bootstrapped, auto or otherwise, that's it.  It is now
> just another node in the cluster.  How it got that way is not
> relevant.
> 
>> At this point we've got 4 nodes in the cluster ... I've gotten this far with no problems,
loaded with tons of data and compared performance with various replication factors. Seeing
faster reads from any particular node (as expected) when the ReplicationFactor is equal to
the number of nodes in the cluster. Have compared lots of single update/creates as well as
batch_mutate (which is very fast for bootstrapping the CFs -- highly recommended).
>> 
>> And also seeing varying performance on reads (fast, and as expected) when ReplicationFactor
< N.
>> 
>> Q3: What, if any issue, is there when R > N?
>> 
> 
> Not recommended.
> 
>> This is the situation as you're bringing up nodes in the cluster. And when you take
down a node (intentionally or as a failure).
>> 
>> I know one consideration is that if R >= N ... and CF data grows ever bigger ...
there will be a hit as the node is created.
>> 
>> Q4: If you know that you're never going to have more than 40 (MaxExpectedClusterNodes)
in your cluster ... is it safe to set R >= MaxExpectedClusterNodes?
>> 
> 
> Setting it higher is not going to help you.  It is also unclear to me
> how having a cluster that large with an RF that high is going to
> behave.  Read repair (which happens on every call) is going to be
> _brutal_.
> 
>> Q5: If you set R = MaxExpectedClusterNodes ... and you end up servicing a node ....
and bringing up an alternate node in its place ... thus having R = N at all times ... and
then you bring up the N+1 node ... will it start to receive the data that it missed while
it was down?
>> 
> 
> This is the Hinted Handoff mechanism.
> 
> 
> b


Mime
View raw message