Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 273E0FEC7 for ; Mon, 29 Apr 2013 09:25:11 +0000 (UTC) Received: (qmail 53157 invoked by uid 500); 29 Apr 2013 09:25:08 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 53116 invoked by uid 500); 29 Apr 2013 09:25:08 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 53106 invoked by uid 99); 29 Apr 2013 09:25:08 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Apr 2013 09:25:08 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a43.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Apr 2013 09:25:02 +0000 Received: from homiemail-a43.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a43.g.dreamhost.com (Postfix) with ESMTP id 82A7E8C058 for ; Mon, 29 Apr 2013 02:24:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=9cRMEGAKi425z3XBWWiBETT6U8 A=; b=XmsGdLUHGmXVP3jl1VadUS7kYop4KVQZjnXwYFuMABuvX1i9njogouHdxn qWaWEk5bdDRcTae0HcmGbQ4mmPXFHbZAUYw9pOKbRDI8Yzb+2yPXIvYJErupgJRw 8isO1CC6KwO3ft3LpcnzN3HHgEpkxXjWaHkZkYa5jCZljJY3o= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a43.g.dreamhost.com (Postfix) with ESMTPSA id 7A3648C057 for ; Mon, 29 Apr 2013 02:24:38 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_FA32AFA4-971A-4214-954F-8E3DD8E5077D" Message-Id: <659187FB-54E3-46CB-B66D-AE4CC4B9DAE2@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Adding nodes in 1.2 with vnodes requires huge disks Date: Mon, 29 Apr 2013 21:24:36 +1200 References: <6BF74559-E40C-46DA-9BD3-5CE27C74C72A@igcorp.com.br> <479C8613-8785-40FC-A4FE-77356483F5FC@thelastpickle.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1503) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_FA32AFA4-971A-4214-954F-8E3DD8E5077D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii is this understanding correct "we had a 12 node cluster with 256 vnodes = on each node (upgraded from 1.1), we added two additional nodes that = streamed so much data (600+Gb when other nodes had 150-200GB) during the = joining phase that they filled their local disks and had to be killed" ? Can you raise a ticket on = https://issues.apache.org/jira/browse/CASSANDRA and update the thread = with the ticket number. Can you show the output from nodetool status so we can get a feel for = the ring? Can you include the logs from one of the nodes that failed to join ?=20 Thanks ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 29/04/2013, at 10:01 AM, John Watson wrote: > On Sun, Apr 28, 2013 at 2:19 PM, aaron morton = wrote: >> We're going to try running a shuffle before adding a new node = again... maybe that will help >=20 > I don't think hurt but I doubt it will help.=20 >=20 > We had to bail on shuffle since we need to add capacity ASAP and not = in 20 days. > =20 >=20 >>> It seems when new nodes join, they are streamed *all* sstables in = the cluster. >=20 >>=20 >=20 > How many nodes did you join, what was the num_tokens ?=20 > Did you notice streaming from all nodes (in the logs) or are you = saying this in response to the cluster load increasing ?=20 >=20 > =20 > Was only adding 2 nodes at the time (planning to add a total of 12.) = Starting with a cluster of 12, but now 11 since 1 node entered some = weird state when one of the new nodes ran out disk space. > num_tokens is set to 256 on all nodes. > Yes, nearly all current nodes were streaming to the new ones (which = was great until disk space was an issue.) >>> The purple line machine, I just stopped the joining process because = the main cluster was dropping mutation messages at this point on a few = nodes (and it still had dozens of sstables to stream.) > Which were the new nodes ? > Can you show the output from nodetool status? >=20 >=20 > The new nodes are the purple and gray lines above all the others. >=20 > nodetool status doesn't show joining nodes. I think I saw a bug = already filed for this but I can't seem to find it. > =20 >=20 > Cheers >=20 > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand >=20 > @aaronmorton > http://www.thelastpickle.com >=20 > On 27/04/2013, at 9:35 AM, Bryan Talbot = wrote: >=20 >> I believe that "nodetool rebuild" is used to add a new datacenter, = not just a new host to an existing cluster. Is that what you ran to add = the node? >>=20 >> -Bryan >>=20 >>=20 >>=20 >> On Fri, Apr 26, 2013 at 1:27 PM, John Watson wrote: >> Small relief we're not the only ones that had this issue. >>=20 >> We're going to try running a shuffle before adding a new node = again... maybe that will help >>=20 >> - John >>=20 >>=20 >> On Fri, Apr 26, 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral = wrote: >> I am using the same version and observed something similar. >>=20 >> I've added a new node, but the instructions from Datastax did not = work for me. Then I ran "nodetool rebuild" on the new node. After = finished this command, it contained two times the load of the other = nodes. Even when I ran "nodetool cleanup" on the older nodes, the = situation was the same. >>=20 >> The problem only seemed to disappear when "nodetool repair" was = applied to all nodes. >>=20 >> Regards, >> Francisco Sobral. >>=20 >>=20 >>=20 >>=20 >> On Apr 25, 2013, at 4:57 PM, John Watson wrote: >>=20 >>> After finally upgrading to 1.2.3 from 1.1.9, enabling vnodes, and = running upgradesstables, I figured it would be safe to start adding = nodes to the cluster. Guess not? >>>=20 >>> It seems when new nodes join, they are streamed *all* sstables in = the cluster. >>>=20 >>> = https://dl.dropbox.com/s/bampemkvlfck2dt/Screen%20Shot%202013-04-25%20at%2= 012.35.24%20PM.png >>>=20 >>> The gray the line machine ran out disk space and for some reason = cascaded into errors in the cluster about 'no host id' when trying to = store hints for it (even though it hadn't joined yet). >>> The purple line machine, I just stopped the joining process because = the main cluster was dropping mutation messages at this point on a few = nodes (and it still had dozens of sstables to stream.) >>>=20 >>> I followed this: = http://www.datastax.com/docs/1.2/operations/add_replace_nodes >>>=20 >>> Is there something missing in that documentation? >>>=20 >>> Thanks, >>>=20 >>> John >>=20 >>=20 >>=20 >=20 >=20 --Apple-Mail=_FA32AFA4-971A-4214-954F-8E3DD8E5077D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii is = this understanding correct "we had a 12 node cluster with 256 vnodes on = each node (upgraded from 1.1), we added two additional nodes that = streamed so much data (600+Gb when other nodes had 150-200GB) during the = joining phase that they filled their local disks and had to be killed" = ?

Can you raise a ticket on https://issues.ap= ache.org/jira/browse/CASSANDRA and update the thread with the = ticket number.

Can you show the output from = nodetool status so we can get a feel for the ring?
Can you = include the logs from one of the nodes that failed to join = ? 

Thanks

http://www.thelastpickle.com

On 29/04/2013, at 10:01 AM, John Watson <john@disqus.com> wrote:

On Sun, Apr 28, 2013 at 2:19 PM, aaron morton <aaron@thelastpickle.com> wrote:
We're going to try running a shuffle before adding a new node = again... maybe that will help
I don't think  hurt but I doubt it will = help. 

We had to bail = on shuffle since we need to add capacity ASAP and not in 20 = days.
 

It seems when new nodes join, they are streamed *all* = sstables in the = cluster.


How many nodes did you join, = what was the num_tokens ? 
Did you notice streaming from all nodes (in the logs) or are you = saying this in response to the cluster load increasing = ? 

<= /span>
 
Was only adding 2 nodes at the time = (planning to add a total of 12.) Starting with a cluster of 12, but now = 11 since 1 node entered some weird state when one of the new nodes ran = out disk space.
num_tokens is set to 256 on all nodes.
Yes, nearly all = current nodes were streaming to the new ones (which was great until disk = space was an issue.)
The purple line machine, I just stopped the joining process = because the main cluster was dropping mutation messages at this point on = a few nodes (and it still had dozens of sstables to stream.)
=
Which were the new nodes ?
Can you show = the output from nodetool status?


The new nodes = are the purple and gray lines above all the = others.

nodetool status doesn't show joining = nodes. I think I saw a bug already filed for this but I can't seem to = find it.
 
=

Cheers

-----------= ------
Aaron Morton
Freelance Cassandra = Consultant
New = Zealand

@aaronmorton

On 27/04/2013, at 9:35 AM, Bryan Talbot <btalbot@aeriagames.com> = wrote:

I believe = that "nodetool rebuild" is used to add a new datacenter, not just a new = host to an existing cluster.  Is that what you ran to add the = node?

-Bryan



On Fri, Apr 26, 2013 at 1:27 PM, John Watson <john@disqus.com> wrote:
Small relief we're not = the only ones that had this issue.

We're going to try running a shuffle before adding a new = node again... maybe that will help

- John


On Fri, Apr 26, = 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral <fsobral@igcorp.com.br> wrote:
I am using the same = version and observed something similar.

I've added a new node, but the instructions from Datastax did not = work for me. Then I ran "nodetool rebuild" on the new node. After = finished this command, it contained two times the load of the other = nodes. Even when I ran "nodetool cleanup" on the older nodes, the = situation was the same.

The problem only seemed to disappear when "nodetool = repair" was applied to all = nodes.

Regards,
Francisco = Sobral.




On Apr 25, 2013, at 4:57 PM, John Watson <john@disqus.com> = wrote:

After finally = upgrading to 1.2.3 from 1.1.9, enabling vnodes, and running = upgradesstables, I figured it would be safe to start adding nodes to the = cluster. Guess not?

It seems when new nodes join, they are streamed *all* = sstables in the cluster.


The gray the line machine ran out disk space = and for some reason cascaded into errors in the cluster about 'no host = id' when trying to store hints for it (even though it hadn't joined = yet).
The purple line machine, I just stopped the joining process because = the main cluster was dropping mutation messages at this point on a few = nodes (and it still had dozens of sstables to stream.)


Is there something missing in that = documentation?

Thanks,

John






= --Apple-Mail=_FA32AFA4-971A-4214-954F-8E3DD8E5077D--