incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Hobbs <>
Subject Re: Virtual Nodes, lots of physical nodes and potentially increasing outage count?
Date Sun, 09 Dec 2012 17:50:16 GMT

Strictly speaking, your math makes the assumption that the failure of
different nodes are probabilistically independent events. This is, of
course, not a accurate assumption for real world conditions.  Nodes share
racks, networking equipment, power, availability zones, data centers, etc.
So, I think the mathematical assertion is not quite as strong as one would
like, but it's certainly a good argument for handling certain types of node

On Fri, Dec 7, 2012 at 11:27 AM, Nicolas Favre-Felix <>wrote:

> Hi Eric,
> Your concerns are perfectly valid.
> We (Acunu) led the design and implementation of this feature and spent a
> long time looking at the impact of such a large change.
> We summarized some of our notes and wrote about the impact of virtual
> nodes on cluster uptime a few months back:
> .
> The main argument in this blog post is that you only have a failure to
> perform quorum read/writes if at least RF replicas fail within the time it
> takes to rebuild the first dead node. We show that virtual nodes actually
> decrease the probability of failure, by streaming data from all nodes and
> thereby improving the rebuild time.
> Regards,
> Nicolas
> On Wed, Dec 5, 2012 at 4:45 PM, Eric Parusel <>wrote:
>> Hi all,
>> I've been wondering about virtual nodes and how cluster uptime might
>> change as cluster size increases.
>> I understand clusters will benefit from increased reliability due to
>> faster rebuild time, but does that hold true for large clusters?
>> It seems that since (and correct me if I'm wrong here) every physical
>> node will likely share some small amount of data with every other node,
>> that as the count of physical nodes in a Cassandra cluster increases (let's
>> say into the triple digits) that the probability of at least one failure to
>> Quorum read/write occurring in a given time period would *increase*.
>> Would this hold true, at least until physical nodes becomes greater than
>> num_tokens per node?
>> I understand that the window of failure for affected ranges would
>> probably be small but we do Quorum reads of many keys, so we'd likely hit
>> every virtual range with our queries, even if num_tokens was 256.
>> Thanks,
>> Eric

Tyler Hobbs
DataStax <>

View raw message