Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B034610DA5 for ; Sun, 21 Dec 2014 15:15:53 +0000 (UTC) Received: (qmail 64554 invoked by uid 500); 21 Dec 2014 15:15:51 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 64513 invoked by uid 500); 21 Dec 2014 15:15:50 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 64503 invoked by uid 99); 21 Dec 2014 15:15:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Dec 2014 15:15:50 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rsvihla@datastax.com designates 209.85.160.177 as permitted sender) Received: from [209.85.160.177] (HELO mail-yk0-f177.google.com) (209.85.160.177) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Dec 2014 15:15:47 +0000 Received: by mail-yk0-f177.google.com with SMTP id 9so1626982ykp.8 for ; Sun, 21 Dec 2014 07:14:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datastax.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=a2L9xuvgnAqumwjEvl7+DdP5gUqp0JrncpHMvWZ143A=; b=eYbFThuWjfJ52zfYgdcXFgdW1wiXkOfzZUimCGXSb8jmTowIE4KuDHelmmS5H59rTR /kQS1xvRd1NtMp5C0LJnPC99fjEWLRtCXScLvjtC9007Z3RrkaZpRovj41WALKdgUcmd 9Bne9YCmhcAZO/Od0FvucunxM2xhX0VbG/tpA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=a2L9xuvgnAqumwjEvl7+DdP5gUqp0JrncpHMvWZ143A=; b=Qk4SVtZ+/bEcgUOY0YP5nz9KjW5JYHyI9Exf6f9Nv7FZsN+l04WT+RU4dV2khG+4aH LLSNzty8Ejhvyt7z+RR3b4OuhfSAkkw4rFs0lSZce+BjuDLV1Ef9yA/y00fV8w4qeJYX vHOTT3NUaxrUkmNlVAE9Hg+vieZ4i2beqUSq7ZOxw3jm8f+Pw4YTC/4T6iHiLyF1XD9X k9RFNpLfHdTOiKHIKqoIN8lZE/gGl1iR05M0H7TlAo+If94RHiE2J67p7zPWB5hVfd6e ID6dz5Y5u5gMFlts1iVrW4ohNWcEhWlCuwjyu6OxCUXDFhSUbGAtT/DSYMyEWgXUCt50 UGzw== X-Gm-Message-State: ALoCoQmdwRivY8Zc9+tZhF9EeRSCCASIVBde5Bqqaa2klsdJoHAvVf3zFUCFUcHz5sgod1TpZV5M MIME-Version: 1.0 X-Received: by 10.170.209.208 with SMTP id a199mr16429221ykf.120.1419174880955; Sun, 21 Dec 2014 07:14:40 -0800 (PST) Received: by 10.170.216.2 with HTTP; Sun, 21 Dec 2014 07:14:40 -0800 (PST) In-Reply-To: References: <5493C8A2.6000503@enercast.de> Date: Sun, 21 Dec 2014 09:14:40 -0600 Message-ID: Subject: Re: Replacing nodes disks From: Ryan Svihla To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113983caa769e8050abb669f X-Virus-Checked: Checked by ClamAV on apache.org --001a113983caa769e8050abb669f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cassandra is designed to rebuild a node from other nodes, whether a node is dead by your hand because you killed it or fate is irrelevant, the process is the same, a "new node" can be the same hostname and ip or it can have totally different ones. On Sun, Dec 21, 2014 at 6:01 AM, Or Sher wrote: > > If I'll use the replace_address parameter with the same IP address, would > that do the job? > > On Sun, Dec 21, 2014 at 11:20 AM, Or Sher wrote: > >> What I want to do is kind of replacing a dead node - >> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations= /ops_replace_node_t.html >> But replacing it with a clean node with the same IP and hostname. >> >> On Sun, Dec 21, 2014 at 9:53 AM, Or Sher wrote: >> >>> Thanks guys. >>> I have to replace all data disks, so I don't have another large enough >>> local disk to move the data to. >>> If I'll have no choice, I will backup the data before on some other nod= e >>> or something, but I'd like to avoid it. >>> I would really love letting Cassandra do it thing and rebuild itself. >>> Did anybody handled such cases that way (Letting Cassandra rebuild it's >>> data?) >>> Although there are no documented procedure for it, It should be possibl= e >>> right? >>> >>> On Fri, Dec 19, 2014 at 8:41 AM, Jan Kesten >>> wrote: >>> >>>> Hi Or, >>>> >>>> I did some sort of this a while ago. If your machines do have a free >>>> disk slot - just put another disk there and use it as another >>>> data_file_directory. >>>> >>>> If not - as in my case: >>>> >>>> - grab an usb dock for disks >>>> - put the new one in there, plug in, format, mount to /mnt etc. >>>> - I did an online rsync from /var/lib/cassandra/data to /mnt >>>> - after that, bring cassandra down >>>> - do another rsync from /var/lib/cassandra/data to /mnt (should be >>>> faster, as sstables do not change, minimizes downtime) >>>> - if you need adjust /etc/fstab if needed >>>> - shutdown the node >>>> - swap disks >>>> - power on the node >>>> - everything should be fine ;-) >>>> >>>> Of course you will need a replication factor > 1 for this to work ;-) >>>> >>>> Just my 2 cents, >>>> Jan >>>> >>>> rsync the full contents there, >>>> >>>> Am 18.12.2014 um 16:17 schrieb Or Sher: >>>> >>>> Hi all, >>>>> >>>>> We have a situation where some of our nodes have smaller disks and we >>>>> would like to align all nodes by replacing the smaller disks to bigge= r ones >>>>> without replacing nodes. >>>>> We don't have enough space to put data on / disk and copy it back to >>>>> the bigger disks so we would like to rebuild the nodes data from othe= r >>>>> replicas. >>>>> >>>>> What do you think should be the procedure here? >>>>> >>>>> I'm guessing it should be something like this but I'm pretty sure it'= s >>>>> not enough. >>>>> 1. shutdown C* node and server. >>>>> 2. replace disks + create the same vg lv etc. >>>>> 3. start C* (Normally?) >>>>> 4. nodetool repair/rebuild? >>>>> *I think I might get some consistency issues for use cases relying on >>>>> Quorum reads and writes for strong consistency. >>>>> What do you say? >>>>> >>>>> Another question is (and I know it depends on many factors but I'd >>>>> like to hear an experienced estimation): How much time would take to >>>>> rebuild a 250G data node? >>>>> >>>>> Thanks in advance, >>>>> Or. >>>>> >>>>> -- >>>>> Or Sher >>>>> >>>> >>>> >>> >>> >>> -- >>> Or Sher >>> >> >> >> >> -- >> Or Sher >> > > > > -- > Or Sher > --=20 [image: datastax_logo.png] Ryan Svihla Solution Architect [image: twitter.png] [image: linkedin.png] DataStax is the fastest, most scalable distributed database technology, delivering Apache Cassandra to the world=E2=80=99s most innovative enterpri= ses. Datastax is built to be agile, always-on, and predictably scalable to any size. With more than 500 customers in 45 countries, DataStax is the database technology and transactional backbone of choice for the worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay. --001a113983caa769e8050abb669f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Cassandra is designed to rebuild a node from other nodes, = whether a node is dead by your hand because you killed it or fate is irrele= vant, the process is the same, a "new node" can be the same hostn= ame and ip or it can have totally different ones.

On Sun, Dec 21, 2014 at 6:01 AM, = Or Sher <or.sher1@gmail.com> wrote:
If I'll use the=C2=A0replace_= address parameter with the same IP address, would that do the job?

On Sun, Dec 21, 2014 at 11:20 AM, Or Sher <<= a href=3D"mailto:or.sher1@gmail.com" target=3D"_blank">or.sher1@gmail.com> wrote:
Wha= t I want to do is kind of replacing a dead node -=C2=A0http://www.datastax.com/documentation/cassa= ndra/2.0/cassandra/operations/ops_replace_node_t.html
But replacing= it with a clean node with the same IP and hostname.

On Sun, Dec 21, 2= 014 at 9:53 AM, Or Sher <or.sher1@gmail.com> wrote:
Thanks guys.
I have to repl= ace all data disks, so I don't have another large enough local disk to = move the data to.
If I'll have no choice, I will backup the d= ata before on some other node or something, but I'd like to avoid it.
I would really love letting Cassandra do it thing and rebuild itse= lf.=C2=A0
Did anybody handled such cases that way (Letting Cassan= dra rebuild it's data?)
Although there are no documented proc= edure for it, It should be possible right?

On Fri, Dec 19, 2014 at 8:41= AM, Jan Kesten <j.kesten@enercast.de> wrote:
Hi Or,

I did some sort of this a while ago. If your machines do have a free disk s= lot - just put another disk there and use it as another data_file_directory= .

If not - as in my case:

- grab an usb dock for disks
- put the new one in there, plug in, format, mount to /mnt etc.
- I did an online rsync from /var/lib/cassandra/data to /mnt
- after that, bring cassandra down
- do another rsync from /var/lib/cassandra/data to /mnt (should be faster, = as sstables do not change, minimizes downtime)
- if you need adjust /etc/fstab if needed
- shutdown the node
- swap disks
- power on the node
- everything should be fine ;-)

Of course you will need a replication factor > 1 for this to work ;-)
Just my 2 cents,
Jan

rsync the full contents there,

Am 18.12.2014 um 16:17 schrieb Or Sher:

Hi all,

We have a situation where some of our nodes have smaller disks and we would= like to align all nodes by replacing the smaller disks to bigger ones with= out replacing nodes.
We don't have enough space to put data on / disk and copy it back to th= e bigger disks so we would like to rebuild the nodes data from other replic= as.

What do you think should be the procedure here?

I'm guessing it should be something like this but I'm pretty sure i= t's not enough.
1. shutdown C* node and server.
2. replace disks + create the same vg lv etc.
3. start C* (Normally?)
4. nodetool repair/rebuild?
*I think I might get some consistency issues for use cases relying on Quoru= m reads and writes for strong consistency.
What do you say?

Another question is (and I know it depends on many factors but I'd like= to hear an experienced estimation): How much time would take to rebuild a = 250G data node?

Thanks in advance,
Or.

--
Or Sher




<= /div>--
Or Sher



<= font color=3D"#888888">--
Or Sher


--
Or Sher

--

Ryan Svihl= a

Solution Architect


3D"twitter.png" = 3D"linkedin.png"=

DataStax is the fastest, most scalable distributed database technol= ogy, delivering Apache Cassandra to the world=E2=80=99s most innovative ent= erprises. Datastax is built to be agile, always-on, and predictably scalabl= e to any size. With more than 500 customers in 45 countries, DataStax is the database technology and tra= nsactional backbone of choice for the worlds most innovative companies such= as Netflix, Adobe, Intuit, and eBay.


=
--001a113983caa769e8050abb669f--