Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of skye.book@gmail.com designates
 209.85.213.180 as permitted sender)
From: Skye Book <skye.book@gmail.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_544AB182-E1D3-4F3D-B688-8B66312F6BA7"
Message-Id: <EB7860D0-596A-489E-BBCB-10A9BB782383@gmail.com>
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Subject: Re: Nodes not added to existing cluster
Date: Wed, 25 Sep 2013 16:12:26 -0400
References: <7174577A-E0A3-4111-8E4E-D466641A38CB@gmail.com>
 <CAEDUwd38KXQn5dn8Lyhh_7FUt7_3RwxaF_csKY=-YXZ2QqV=3A@mail.gmail.com>
To: user@cassandra.apache.org
In-Reply-To: 
 <CAEDUwd38KXQn5dn8Lyhh_7FUt7_3RwxaF_csKY=-YXZ2QqV=3A@mail.gmail.com>


--Apple-Mail=_544AB182-E1D3-4F3D-B688-8B66312F6BA7
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1

Thank you, both Michael and Robert for your suggestions.  I actually saw =
5760, but we were running on 2.0.0, which it seems like this was fixed =
in.

That said, I noticed that my Chef scripts were failing to set the =
broadcast_address correctly, which I'm guessing is the cause of the =
problem, fixing that and trying a redeploy.  I am curious, though, how =
any of this worked in the first place spread across three AZ's without =
that being set?

-Skye

On Sep 25, 2013, at 3:56 PM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Wed, Sep 25, 2013 at 12:41 PM, Skye Book <skye.book@gmail.com> =
wrote:
> I have a three node cluster using the EC2 Multi-Region Snitch =
currently operating only in US-EAST.  On having a node go down this =
morning, I started a new node with an identical configuration, except =
for the seed list, the listen address and the rpc address.  The new node =
comes up and creates its own cluster rather than joining the =
pre-existing ring.  I've tried creating a node both before ad after =
using `nodetool remove` for the bad node, each time with the same =
result.
>=20
> What version of Cassandra?
>=20
> This particular confusing behavior is fixed upstream, in a version you =
should not deploy to production yet. Take some solace, however, that you =
may be the last Cassandra administrator to die for a broken code path!
>=20
> https://issues.apache.org/jira/browse/CASSANDRA-5768
>=20
> Does anyone have any suggestions for where to look that might put me =
on the right track?
>=20
> It must be that your seed list is wrong in some way, or your node =
state is wrong. If you're trying to bootstrap a node, note that you =
can't bootstrap a node when it is in its own seed list.
>=20
> If you have installed Cassandra via debian package, there is a =
possibility that your node has started before you explicitly started it. =
If so, it might have invalid node state.
>=20
> Have you tried wiping the data directory and trying again?
>=20
> What is your seed list? Are you sure the new node can reach the seeds =
on the network layer?
>=20
> =3DRob


--Apple-Mail=_544AB182-E1D3-4F3D-B688-8B66312F6BA7
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=iso-8859-1

<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Thank you, both Michael and Robert for your suggestions. &nbsp;I actually saw 5760, but we were running on 2.0.0, which it seems like this was fixed in.<div><br></div><div>That said, I noticed that my Chef scripts were failing to set the broadcast_address correctly, which I'm guessing is the cause of the problem, fixing that and trying a redeploy. &nbsp;I am curious, though, how any of this worked in the first place spread across three AZ's without that being set?</div><div><br><div apple-content-edited="true">
<div><div><div style="margin: 0px; "><font face="Helvetica, sans-serif">-Skye</font></div></div></div>
</div>
<br><div><div>On Sep 25, 2013, at 3:56 PM, Robert Coli &lt;<a href="mailto:rcoli@eventbrite.com">rcoli@eventbrite.com</a>&gt; wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">On Wed, Sep 25, 2013 at 12:41 PM, Skye Book <span dir="ltr">&lt;<a href="mailto:skye.book@gmail.com" target="_blank">skye.book@gmail.com</a>&gt;</span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>I have a three node cluster using the EC2 Multi-Region Snitch currently operating only in US-EAST. &nbsp;On having a node go down this morning, I started a new node with an identical configuration, except for the seed list, the listen address and the rpc address. &nbsp;The new node comes up and creates its own cluster rather than joining the pre-existing ring. &nbsp;I've tried creating a node both <i>before</i>&nbsp;ad <i>after</i>&nbsp;using `nodetool remove` for the bad node, each time with the same result.</div>
</div></blockquote><div><br></div><div>What version of Cassandra?</div><div><br></div><div>This particular confusing behavior is fixed upstream, in a version you should not deploy to production yet. Take some solace, however, that you may be the last Cassandra administrator to die for a broken code path!</div>
<div><span style="white-space: pre-wrap; "><br></span></div><div><a href="https://issues.apache.org/jira/browse/CASSANDRA-5768">https://issues.apache.org/jira/browse/CASSANDRA-5768</a><br></div><div><br></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Does anyone have any suggestions for where to look that might put me on the right track?</div>
</div></blockquote><div><br></div><div>It must be that your seed list is wrong in some way, or your node state is wrong. If you're trying to bootstrap a node, note that you can't bootstrap a node when it is in its own seed list.</div>
<div><br></div><div>If you have installed Cassandra via debian package, there is a possibility that your node has started before you explicitly started it. If so, it might have invalid node state.</div><div><br></div><div>
Have you tried wiping the data directory and trying again?</div><div><br></div><div>What is your seed list? Are you sure the new node can reach the seeds on the network layer?</div><div><br></div><div>=Rob</div></div></div>
</div>
</blockquote></div><br></div></body></html>
--Apple-Mail=_544AB182-E1D3-4F3D-B688-8B66312F6BA7--