Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of soverton@acunu.com designates
 209.85.223.180 as permitted sender)
MIME-Version: 1.0
Reply-To: sam@acunu.com
In-Reply-To: 
 <CA+qt5VM9-p-GUVS6mv=r5FGrdu5PL+kfdbuQX2Rphv527c-3Wg@mail.gmail.com>
References: 
 <CA+qt5VM9-p-GUVS6mv=r5FGrdu5PL+kfdbuQX2Rphv527c-3Wg@mail.gmail.com>
From: Sam Overton <sam@acunu.com>
Date: Mon, 29 Apr 2013 12:08:57 +0100
Message-ID: 
 <CADjM4ztNKLR1HM_Cx7qdo0Q8+pyoqjaOC-gtHVgCOi6=_bE8QA@mail.gmail.com>
Subject: Re: cassandra-shuffle time to completion and required disk space
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=e89a8f3b9c2373a04604db7dea5f

--e89a8f3b9c2373a04604db7dea5f
Content-Type: text/plain; charset=ISO-8859-1

An alternative to running shuffle is to do a rolling
bootstrap/decommission. You would set num_tokens on the existing hosts (and
restart them) so that they split their ranges, then bootstrap in N new
hosts, then decommission the old ones.


On 28 April 2013 22:21, John Watson <john@disqus.com> wrote:

> The amount of time/space cassandra-shuffle requires when upgrading to
> using vnodes should really be apparent in documentation (when some is made).
>
> Only semi-noticeable remark about the exorbitant amount of time is a
> bullet point in: http://wiki.apache.org/cassandra/VirtualNodes/Balance
>
> "Shuffling will entail moving a lot of data around the cluster and so has
> the potential to consume a lot of disk and network I/O, and to take a
> considerable amount of time. For this to be an online operation, the
> shuffle will need to operate on a lower priority basis to other streaming
> operations, and should be expected to take days or weeks to complete."
>
> We tried running shuffle on a QA version of our cluster and 2 things were
> brought to light:
>  - Even with no reads/writes it was going to take 20 days
>  - Each machine needed enough free diskspace to potentially hold the
> entire cluster's sstables on disk
>
> Regards,
>
> John
>


-- 
Sam Overton
Acunu | http://www.acunu.com | @acunu

--e89a8f3b9c2373a04604db7dea5f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">An alternative to running shuffle is to do a rolling boots=
trap/decommission. You would set num_tokens on the existing hosts (and rest=
art them) so that they split their ranges, then bootstrap in N new hosts, t=
hen=A0decommission=A0the old ones.<div>

<br></div></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quot=
e">On 28 April 2013 22:21, John Watson <span dir=3D"ltr">&lt;<a href=3D"mai=
lto:john@disqus.com" target=3D"_blank">john@disqus.com</a>&gt;</span> wrote=
:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<div dir=3D"ltr">The amount of time/space cassandra-shuffle requires when u=
pgrading to using vnodes should really be apparent in documentation (when s=
ome is made).<div><br></div><div>Only semi-noticeable remark about the exor=
bitant amount of time is a bullet point in:=A0<a href=3D"http://wiki.apache=
.org/cassandra/VirtualNodes/Balance" target=3D"_blank">http://wiki.apache.o=
rg/cassandra/VirtualNodes/Balance</a></div>


<div><br></div><div>&quot;Shuffling will entail moving a lot of data around=
 the cluster and so has the potential to consume a lot of disk and network =
I/O, and to take a considerable amount of time. For this to be an online op=
eration, the shuffle will need to operate on a lower priority basis to othe=
r streaming operations, and should be expected to take days or weeks to com=
plete.&quot;</div>


<div><br></div><div>We tried running shuffle on a QA version of our cluster=
 and 2 things were brought to light:</div><div>=A0- Even with no reads/writ=
es it was going to take 20 days</div><div>=A0- Each machine needed enough f=
ree diskspace to potentially hold the entire cluster&#39;s sstables on disk=
</div>


<div><br></div><div>Regards,</div><div><br></div><div>John</div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><span style=
=3D"color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px">Sam=
 Overton<br>Acunu |=A0<a href=3D"http://www.acunu.com/" style=3D"color:rgb(=
0,0,204)" target=3D"_blank">http://www.acunu.com</a>=A0| @acunu</span>
</div>

--e89a8f3b9c2373a04604db7dea5f--