Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DDBCA10417 for ; Fri, 26 Apr 2013 14:34:33 +0000 (UTC) Received: (qmail 4497 invoked by uid 500); 26 Apr 2013 14:34:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 4469 invoked by uid 500); 26 Apr 2013 14:34:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 4461 invoked by uid 99); 26 Apr 2013 14:34:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Apr 2013 14:34:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dmcnelis@gmail.com designates 209.85.217.180 as permitted sender) Received: from [209.85.217.180] (HELO mail-lb0-f180.google.com) (209.85.217.180) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Apr 2013 14:34:27 +0000 Received: by mail-lb0-f180.google.com with SMTP id t11so3867771lbi.11 for ; Fri, 26 Apr 2013 07:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=M+nMcWN5Z7OIv8EBeG1U1WDXHOuAFu3JioSch065jTc=; b=EENIuP4GxWkUxHj62xr4P6DQGLB9g/OpFv6LK5qoVM2NGHn+ByGWSX3DwKRMT1zYym IvP3js9aQDmV1CeOyFnEADJzTZiBrQ5CtnY3enbltBUe5JXf8pMBRdaZMJh30PTRTwSc IuAvcIXeQ/MjPFQG1eWdh21ZK/s4G8ZtNjHqvdOWjgv248PRq2UHmUD7SdyuQADEHC0v sVJTwC8OmUH1szC0fwzwyG1hqxKbm7yR+EC9RRNaIwFIN9SCF38k1pxyK9ZXh4TcQkAQ JLdNCqjG8tKvBwNf+LLYCy0jwvyAadLqYzmKUZ3Q0u70VQdnmVDHSMwixHka77uAElRf dWxw== MIME-Version: 1.0 X-Received: by 10.152.3.137 with SMTP id c9mr22657184lac.5.1366986845701; Fri, 26 Apr 2013 07:34:05 -0700 (PDT) Received: by 10.112.173.136 with HTTP; Fri, 26 Apr 2013 07:34:05 -0700 (PDT) In-Reply-To: References: Date: Fri, 26 Apr 2013 09:34:05 -0500 Message-ID: Subject: Re: vnodes and load balancing - 1.2.4 From: David McNelis To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e0149423259cd5b04db446d9a X-Virus-Checked: Checked by ClamAV on apache.org --089e0149423259cd5b04db446d9a Content-Type: text/plain; charset=ISO-8859-1 Decommissioning those nodes isn't a problem. When you say remove all the data, I assume you mean rm -rf my data directory (the default /var/lib/cassandra/data I'd done this prior to starting up the nodes, because they were installed with from the apt-get repo, which automatically starts cassandra (bad form, on that, as a side not). But the first time I tried to start the node after setting my config, I got an error that System.Users didn't exist and exited out. Second time I tried to start the nodes, they started. Outside of clearing the data, having no value for initial token, having num_tokens set, is there anything else I need to do to bring them in and bootstrap them? Note, this isn't the first nodes that I've added to the cluster, but they are giving me fits. Additionally, this morning, my seed nodes were all flipping out with an error like: https://gist.github.com/dmcnelis/5467636 (AssertionError, when trying to determine ranges for nodes) Once I decommissioned the new nodes, I had no more errors in my seed node logs. On Fri, Apr 26, 2013 at 5:48 AM, Sam Overton wrote: > Some extra information you could provide which will help debug this: the > logs from those 3 nodes which have no data and the output of "nodetool ring" > > Before seeing those I can only guess, but my guess would be that in the > logs on those 3 nodes you will see this: "Calculating new tokens" and this: > "Split previous range (blah, blah] into " > > If that is the case then it means you accidentally started those three > nodes with the default configuration (single-token) and then subsequently > changed (num_tokens) and then joined them into the cluster. What happens > when you do this is that the node thinks it used to be responsible for a > single range and is being migrated to vnodes, so it splits its single range > (now a very small part of the keyspace) into 256 smaller ranges, and ends > up with just a tiny portion of the ring assigned to it. > > To fix this you'll need to decommission those 3 nodes, remove all data > from them, then bootstrap them in again with the correct configuration from > the start. > > Sam > > > > On 26 April 2013 06:07, David McNelis wrote: > >> So, I had 7 nodes that I set up using vnodes, 256 tokens each, no problem. >> >> I added two 512 token nodes, no problem, things seemed to balance. >> >> The next 3 nodes I added, all at 256 tokens, and they have a cumulative >> load of 116mb (where as the other nodes are at ~100GB and ~200GB (256 and >> 512 respectively). >> >> Anyone else seen this is 1.2.4? >> >> The nodes seem to join the cluster ok, and I have num_tokens set and have >> tried both an empty initial_token and a commented out initial token, with >> no change. >> >> I see nothing streaming with netstats either, though these nodes were >> added days apart. At first I thought I must have a hot key or something, >> but that doesn't seem to be the case, since the node I thought that one was >> on has evened out over the past couple of days with no new nodes added. >> >> I really *DON'T* want to deal with another shuffle....but what options do >> I have, since vnodes "make it unneeded to balance the cluster"? (which, at >> the moment, seems like a load of bullshit). >> > > > > -- > Sam Overton > Acunu | http://www.acunu.com | @acunu > --089e0149423259cd5b04db446d9a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Decommissioning those nodes isn't a problem. =A0When y= ou say remove all the data, I assume you mean rm -rf my data directory (the= default /var/lib/cassandra/data

I'd done this= prior to starting up the nodes, because they were installed with from the = apt-get repo, which automatically starts cassandra (bad form, on that, as a= side not). =A0But the first time I tried to start the node after setting m= y config, I got an error that System.Users didn't exist and exited out.= =A0Second time I tried to start the nodes, they started.

Outside of clearing the data, having no val= ue for initial token, having num_tokens set, is there anything else I need = to do to bring them in and bootstrap them? =A0

Note, this isn't the first nodes that I've added to the = cluster, but they are giving me fits. =A0Additionally, this morning, my see= d nodes were all flipping out with an error like:=A0
https://gist.github.com/dmcn= elis/5467636 =A0(AssertionError, when trying to determine ranges for no= des)

Once I decommissioned the new nodes, = I had no more errors in my seed node logs.


On Fri, Apr 26, 2013 at 5:48 AM, S= am Overton <sam@acunu.com> wrote:
Some extra information you = could provide which will help debug this: the logs from those 3 nodes which= have no data and the output of "nodetool ring"

Before seeing those I can only guess, but my guess would be = that in the logs on those 3 nodes you will see this: "Calculating new = tokens" and this: "Split previous range (blah, blah] into <lon= g list of tokens>"

If that is the case then it means you accidentally star= ted those three nodes with the default configuration (single-token) and the= n subsequently changed (num_tokens) and then joined them into the cluster. = What happens when you do this is that the node thinks it used to be respons= ible for a single range and is being migrated to vnodes, so it splits its s= ingle range (now a very small part of the keyspace) into 256 smaller ranges= , and ends up with just a tiny portion of the ring assigned to it.

To fix this you'll need to decommission those 3 nod= es, remove all data from them, then bootstrap them in again with the correc= t configuration from the start.

Sam

<= br>
On 26 April 2013 06:07, David McNelis <= dmcnelis@gmail.com> wrote:
So, I had 7 nodes that I se= t up using vnodes, 256 tokens each, no problem.

I added = two 512 token nodes, no problem, things seemed to balance.

The next 3 nodes I added, all at 256 tokens, and they h= ave a cumulative load of 116mb (where as the other nodes are at ~100GB and = ~200GB (256 and 512 respectively). =A0

Anyone else seen this is 1.2.4?

The nodes seem to join the cluster ok, and I have num_tokens set and have= tried both an empty initial_token and a commented out initial token, with = no change.

I see nothing streaming with netstats either, though th= ese nodes were added days apart. =A0At first I thought I must have a hot ke= y or something, but that doesn't seem to be the case, since the node I = thought that one was on has evened out over the past couple of days with no= new nodes added.

I really *DON'T* want to deal with another shuffle.= ...but what options do I have, since vnodes "make it unneeded to balan= ce the cluster"? =A0(which, at the moment, seems like a load of bullsh= it).



--
Sam Overton
Acunu= |=A0http://www.acunu.com=A0| @acunu
--089e0149423259cd5b04db446d9a--