incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Hobbs <ty...@datastax.com>
Subject Re: Virtual Nodes, lots of physical nodes and potentially increasing outage count?
Date Sun, 09 Dec 2012 17:50:16 GMT
Nicolas,

Strictly speaking, your math makes the assumption that the failure of
different nodes are probabilistically independent events. This is, of
course, not a accurate assumption for real world conditions.  Nodes share
racks, networking equipment, power, availability zones, data centers, etc.
So, I think the mathematical assertion is not quite as strong as one would
like, but it's certainly a good argument for handling certain types of node
failures.


On Fri, Dec 7, 2012 at 11:27 AM, Nicolas Favre-Felix <nicolas@acunu.com>wrote:

> Hi Eric,
>
> Your concerns are perfectly valid.
>
> We (Acunu) led the design and implementation of this feature and spent a
> long time looking at the impact of such a large change.
> We summarized some of our notes and wrote about the impact of virtual
> nodes on cluster uptime a few months back:
> http://www.acunu.com/2/post/2012/10/improving-cassandras-uptime-with-virtual-nodes.html
> .
> The main argument in this blog post is that you only have a failure to
> perform quorum read/writes if at least RF replicas fail within the time it
> takes to rebuild the first dead node. We show that virtual nodes actually
> decrease the probability of failure, by streaming data from all nodes and
> thereby improving the rebuild time.
>
> Regards,
>
> Nicolas
>
>
> On Wed, Dec 5, 2012 at 4:45 PM, Eric Parusel <ericparusel@gmail.com>wrote:
>
>> Hi all,
>>
>> I've been wondering about virtual nodes and how cluster uptime might
>> change as cluster size increases.
>>
>> I understand clusters will benefit from increased reliability due to
>> faster rebuild time, but does that hold true for large clusters?
>>
>> It seems that since (and correct me if I'm wrong here) every physical
>> node will likely share some small amount of data with every other node,
>> that as the count of physical nodes in a Cassandra cluster increases (let's
>> say into the triple digits) that the probability of at least one failure to
>> Quorum read/write occurring in a given time period would *increase*.
>>
>> Would this hold true, at least until physical nodes becomes greater than
>> num_tokens per node?
>>
>> I understand that the window of failure for affected ranges would
>> probably be small but we do Quorum reads of many keys, so we'd likely hit
>> every virtual range with our queries, even if num_tokens was 256.
>>
>> Thanks,
>> Eric
>>
>
>


-- 
Tyler Hobbs
DataStax <http://datastax.com/>

Mime
View raw message