incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <>
Subject Re: cassandra-shuffle time to completion and required disk space
Date Tue, 30 Apr 2013 20:03:43 GMT
> These are taken just before starting shuffle (ran repair/cleanup the day before).
> During shuffle disabled all reads/writes to the cluster.
> nodetool status keyspace:
> Load       Tokens  Owns (effective)  Host ID
> 80.95 GB   256     16.7%             754f9f4c-4ba7-4495-97e7-1f5b6755cb27

I'm a little confused when nodetool status was showing 256 tokens before the shuffle was run.

Did you set num_tokens during the upgrade process ? But I doubt that would change anything
as the inital_token is set in the system tables during bootstrap. 

`bin/cassandra-shuffle  ls` will show the list of moves the shuffle process was/is going to
run. What does that say?


Aaron Morton
Freelance Cassandra Consultant
New Zealand


On 30/04/2013, at 5:08 AM, John Watson <> wrote:

> That's what we tried first before the shuffle. And ran into the space issue.
> That's detailed in another thread title: "Adding nodes in 1.2 with vnodes requires huge
> On Mon, Apr 29, 2013 at 4:08 AM, Sam Overton <> wrote:
> An alternative to running shuffle is to do a rolling bootstrap/decommission. You would
set num_tokens on the existing hosts (and restart them) so that they split their ranges, then
bootstrap in N new hosts, then decommission the old ones.
> On 28 April 2013 22:21, John Watson <> wrote:
> The amount of time/space cassandra-shuffle requires when upgrading to using vnodes should
really be apparent in documentation (when some is made).
> Only semi-noticeable remark about the exorbitant amount of time is a bullet point in:
> "Shuffling will entail moving a lot of data around the cluster and so has the potential
to consume a lot of disk and network I/O, and to take a considerable amount of time. For this
to be an online operation, the shuffle will need to operate on a lower priority basis to other
streaming operations, and should be expected to take days or weeks to complete."
> We tried running shuffle on a QA version of our cluster and 2 things were brought to
>  - Even with no reads/writes it was going to take 20 days
>  - Each machine needed enough free diskspace to potentially hold the entire cluster's
sstables on disk
> Regards,
> John
> -- 
> Sam Overton
> Acunu | | @acunu

View raw message