incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: cassandra-shuffle time to completion and required disk space
Date Tue, 30 Apr 2013 20:03:43 GMT
> These are taken just before starting shuffle (ran repair/cleanup the day before).
> During shuffle disabled all reads/writes to the cluster.
> 
> nodetool status keyspace:
> 
> Load       Tokens  Owns (effective)  Host ID
> 80.95 GB   256     16.7%             754f9f4c-4ba7-4495-97e7-1f5b6755cb27

I'm a little confused when nodetool status was showing 256 tokens before the shuffle was run.


Did you set num_tokens during the upgrade process ? But I doubt that would change anything
as the inital_token is set in the system tables during bootstrap. 

`bin/cassandra-shuffle  ls` will show the list of moves the shuffle process was/is going to
run. What does that say?

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 30/04/2013, at 5:08 AM, John Watson <john@disqus.com> wrote:

> That's what we tried first before the shuffle. And ran into the space issue.
> 
> That's detailed in another thread title: "Adding nodes in 1.2 with vnodes requires huge
disks"
> 
> 
> On Mon, Apr 29, 2013 at 4:08 AM, Sam Overton <sam@acunu.com> wrote:
> An alternative to running shuffle is to do a rolling bootstrap/decommission. You would
set num_tokens on the existing hosts (and restart them) so that they split their ranges, then
bootstrap in N new hosts, then decommission the old ones.
> 
> 
> 
> On 28 April 2013 22:21, John Watson <john@disqus.com> wrote:
> The amount of time/space cassandra-shuffle requires when upgrading to using vnodes should
really be apparent in documentation (when some is made).
> 
> Only semi-noticeable remark about the exorbitant amount of time is a bullet point in:
http://wiki.apache.org/cassandra/VirtualNodes/Balance
> 
> "Shuffling will entail moving a lot of data around the cluster and so has the potential
to consume a lot of disk and network I/O, and to take a considerable amount of time. For this
to be an online operation, the shuffle will need to operate on a lower priority basis to other
streaming operations, and should be expected to take days or weeks to complete."
> 
> We tried running shuffle on a QA version of our cluster and 2 things were brought to
light:
>  - Even with no reads/writes it was going to take 20 days
>  - Each machine needed enough free diskspace to potentially hold the entire cluster's
sstables on disk
> 
> Regards,
> 
> John
> 
> 
> 
> -- 
> Sam Overton
> Acunu | http://www.acunu.com | @acunu
> 


Mime
View raw message