From user-return-36817-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu Sep 26 04:34:46 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 405E51051B for ; Thu, 26 Sep 2013 04:34:46 +0000 (UTC) Received: (qmail 74866 invoked by uid 500); 26 Sep 2013 04:34:41 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 74809 invoked by uid 500); 26 Sep 2013 04:34:40 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 74796 invoked by uid 99); 26 Sep 2013 04:34:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Sep 2013 04:34:37 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.160.51] (HELO mail-pb0-f51.google.com) (209.85.160.51) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Sep 2013 04:34:31 +0000 Received: by mail-pb0-f51.google.com with SMTP id jt11so584932pbb.24 for ; Wed, 25 Sep 2013 21:34:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=6XGaj35kyZx5wxLVAURGFYgr6CQkwiDx0tLEH6H28nA=; b=VtpQroyqHjRGzhV8NYEmp/GcWHBN/gJu7seqDU876OvAAf6y3zdp00R95ty/X1CvUM rRB/Zbl5BblYKkOl6palYKbQUF4tecy3g2O+Kbojcd+ySqv6TEr2WCYd8Rvw4drI8qDr Kms13AVd82zLIR6Vh2R4bmtE3yokuoQzn9080HZJI7G+rKSiNREg4mvSNi0AshYeHWiH 7cXl5P4EQLdvbx0UZHjHodYL386QbU72yKwlU4JzQ5n9FbaGamuZ+UMjkG//Ffhde0DJ EkPDZyc0ioHSJp3bPG06A2dJMtxva7S5iBczS5NVtAaz0nGyha6hDlvvgcFHIhjB40KL wo8w== X-Gm-Message-State: ALoCoQmFaXbSNGpsofl7CnhNB/5P1d09sBXisn92+ih1fzlyoCNQ3u/OQuFXuNTk68UjylJrcdIs X-Received: by 10.68.200.100 with SMTP id jr4mr37588007pbc.0.1380170049755; Wed, 25 Sep 2013 21:34:09 -0700 (PDT) Received: from [172.16.1.18] ([203.86.207.101]) by mx.google.com with ESMTPSA id dk3sm35884784pbc.32.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 25 Sep 2013 21:34:08 -0700 (PDT) From: Aaron Morton Content-Type: multipart/alternative; boundary="Apple-Mail=_59A146D0-9828-4A93-9648-599A299D502F" Message-Id: <2A167C05-8C7B-4FF1-A0EC-F8B9A1128D91@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) Subject: Re: Nodes not added to existing cluster Date: Thu, 26 Sep 2013 16:34:05 +1200 References: <7174577A-E0A3-4111-8E4E-D466641A38CB@gmail.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1510) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_59A146D0-9828-4A93-9648-599A299D502F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > I am curious, though, how any of this worked in the first place = spread across three AZ's without that being set? boradcast_address is only needed when you are going cross region (IIRC = it's the EC2MultiRegionSnitch) that sets it.=20 As rob said, make sure the seed list includes on of the other nodes and = that the cluster_name set.=20 Cheers ----------------- Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 26/09/2013, at 8:12 AM, Skye Book wrote: > Thank you, both Michael and Robert for your suggestions. I actually = saw 5760, but we were running on 2.0.0, which it seems like this was = fixed in. >=20 > That said, I noticed that my Chef scripts were failing to set the = broadcast_address correctly, which I'm guessing is the cause of the = problem, fixing that and trying a redeploy. I am curious, though, how = any of this worked in the first place spread across three AZ's without = that being set? >=20 > -Skye >=20 > On Sep 25, 2013, at 3:56 PM, Robert Coli wrote: >=20 >> On Wed, Sep 25, 2013 at 12:41 PM, Skye Book = wrote: >> I have a three node cluster using the EC2 Multi-Region Snitch = currently operating only in US-EAST. On having a node go down this = morning, I started a new node with an identical configuration, except = for the seed list, the listen address and the rpc address. The new node = comes up and creates its own cluster rather than joining the = pre-existing ring. I've tried creating a node both before ad after = using `nodetool remove` for the bad node, each time with the same = result. >>=20 >> What version of Cassandra? >>=20 >> This particular confusing behavior is fixed upstream, in a version = you should not deploy to production yet. Take some solace, however, that = you may be the last Cassandra administrator to die for a broken code = path! >>=20 >> https://issues.apache.org/jira/browse/CASSANDRA-5768 >>=20 >> Does anyone have any suggestions for where to look that might put me = on the right track? >>=20 >> It must be that your seed list is wrong in some way, or your node = state is wrong. If you're trying to bootstrap a node, note that you = can't bootstrap a node when it is in its own seed list. >>=20 >> If you have installed Cassandra via debian package, there is a = possibility that your node has started before you explicitly started it. = If so, it might have invalid node state. >>=20 >> Have you tried wiping the data directory and trying again? >>=20 >> What is your seed list? Are you sure the new node can reach the seeds = on the network layer? >>=20 >> =3DRob >=20 --Apple-Mail=_59A146D0-9828-4A93-9648-599A299D502F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1  I am curious, though, how any of this = worked in the first place spread across three AZ's without that being = set?
http://www.thelastpickle.com

On 26/09/2013, at 8:12 AM, Skye Book <skye.book@gmail.com> = wrote:

Thank = you, both Michael and Robert for your suggestions.  I actually saw = 5760, but we were running on 2.0.0, which it seems like this was fixed = in.

That said, I noticed that my Chef scripts were = failing to set the broadcast_address correctly, which I'm guessing is = the cause of the problem, fixing that and trying a redeploy.  I am = curious, though, how any of this worked in the first place spread across = three AZ's without that being set?

-Skye

On Sep 25, 2013, at 3:56 PM, Robert Coli <rcoli@eventbrite.com> = wrote:

On Wed, Sep 25, 2013 at 12:41 PM, Skye = Book <skye.book@gmail.com> wrote:
I have a three node = cluster using the EC2 Multi-Region Snitch currently operating only in = US-EAST.  On having a node go down this morning, I started a new = node with an identical configuration, except for the seed list, the = listen address and the rpc address.  The new node comes up and = creates its own cluster rather than joining the pre-existing ring. =  I've tried creating a node both before ad = after using `nodetool remove` for the bad node, each time = with the same result.

What version of = Cassandra?

This particular confusing behavior = is fixed upstream, in a version you should not deploy to production yet. = Take some solace, however, that you may be the last Cassandra = administrator to die for a broken code path!


Does anyone have any suggestions for = where to look that might put me on the right track?

It must be that your seed list is = wrong in some way, or your node state is wrong. If you're trying to = bootstrap a node, note that you can't bootstrap a node when it is in its = own seed list.

If you have installed Cassandra via debian package, = there is a possibility that your node has started before you explicitly = started it. If so, it might have invalid node = state.

Have you tried wiping the data directory and trying = again?

What is your seed list? Are you sure the = new node can reach the seeds on the network = layer?

=3DRob


= --Apple-Mail=_59A146D0-9828-4A93-9648-599A299D502F--