Hi Theo,

The number (let's call it T and the number of nodes N) 256 was chosen to give good load balancing for random token assignments for most cluster sizes.  For small T, a random choice of initial tokens will in most cases give a poor distribution of data.  The larger T is, the closer to uniform the distribution will be, with increasing probability.

Also, for small T, when a new node is added, it won't have many ranges to split so won't be able to take an even slice of the data.

For this reason T should be large.  But if it is too large, there are too many slices to keep track of as you say.  The function to find which keys live where becomes more expensive and operations that deal with individual vnodes e.g. repair become slow.  (An extreme example is SELECT * LIMIT 1, which when there is no data has to scan each vnode in turn in search of a single row.  This is O(NT) and for even quite small T takes seconds to complete.)

So 256 was chosen to be a reasonable balance.  I don't think most users will find it too slow; users with extremely large clusters may need to increase it.


On 10 June 2013 18:55, Theo Hultberg <theo@iconara.net> wrote:
I'm not sure I follow what you mean, or if I've misunderstood what Cassandra is telling me. Each node has 256 vnodes (or tokens, as the prefered name seems to be). When I run `nodetool status` each node is reported as having 256 vnodes, regardless of how many nodes are in the cluster. A single node cluster has 256 vnodes on the single node, a six node cluster has 256 nodes on each machine, making 1590 vnodes in total. When I run `SELECT tokens FROM system.peers` or `nodetool ring` each node lists 256 tokens.

This is different from how it works in Riak and Voldemort, if I'm not mistaken, and that is the source of my confusion.


On Mon, Jun 10, 2013 at 4:54 PM, Milind Parikh <milindparikh@gmail.com> wrote:

There are n vnodes regardless of the size of the physical cluster.

On Jun 10, 2013 7:48 AM, "Theo Hultberg" <theo@iconara.net> wrote:

The default number of vnodes is 256, is there any significance in this number? Since Cassandra's vnodes don't work like for example Riak's, where there is a fixed number of vnodes distributed evenly over the nodes, why so many? Even with a moderately sized cluster you get thousands of slices. Does this matter? If your cluster grows to over thirty machines and you start looking at ten thousand slices, would that be a problem? I guess trat traversing a list of a thousand or ten thousand slices to find where a token lives isn't a huge problem, but are there any other up or downsides to having a small or large number of vnodes per node?

I understand the benefits for splitting up the ring into pieces, for example to be able to stream data from more nodes when bootstrapping a new one, but that works even if each node only has say 32 vnodes (unless your cluster is truly huge).