Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of driftx@gmail.com designates
 209.85.214.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACjoVsKhaTeMU7g+5HSoWYwYLWgd3+47RwfEJ6Xxp3193jwqqw@mail.gmail.com>
References: 
 <CACjoVsKn3NGXeNyf0EZ0i8zf4AAppkg7HetaRJsj1UpF_gjncg@mail.gmail.com>
 <CACjoVsJK+BZTWG5ruMBsYY6BBRHG_xWUcTSdC2_2p3Vv+1rYqA@mail.gmail.com>
 <CALdd-zhfZGNZaHSjEv79JVryk+2h=o55hbwbAKiuo03Lqmqqng@mail.gmail.com>
 <CACjoVs+R7mm+rnq-+KuToiNCkQ7RAbn_h745YpGvFWocOQXxWg@mail.gmail.com>
 <CAOT3TWqDEu6eyrCokdMrcAd73nJaUJdDXCqOfFsQadOJBFeNvQ@mail.gmail.com>
 <CACjoVsKhaTeMU7g+5HSoWYwYLWgd3+47RwfEJ6Xxp3193jwqqw@mail.gmail.com>
From: Brandon Williams <driftx@gmail.com>
Date: Thu, 13 Oct 2011 09:22:47 -0500
Message-ID: 
 <CAAafH9RbFDdSBrQqcfqshX+gT0doHmuhcCp4dnSzxk6fDOu6LQ@mail.gmail.com>
Subject: Re: Schema versions reflect schemas on unwanted nodes
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You're running into https://issues.apache.org/jira/browse/CASSANDRA-3259

Try upgrading and doing a rolling restart.

-Brandon

On Thu, Oct 13, 2011 at 9:11 AM, Eric Czech <eric@nextbigsound.com> wrote:
> Nope, there was definitely no intersection of the seed nodes between the =
two
> clusters so I'm fairly certain that the second cluster found out about th=
e
> first through what was in the LocationInfo* system tables. =A0Also, I don=
't
> think that procedure will really help because I don't actually want the
> schema on cass-analysis-1 to be consistent with the schema in the origina=
l
> cluster -- I just want to totally remove it.
>
> On Thu, Oct 13, 2011 at 8:01 AM, Mohit Anchlia <mohitanchlia@gmail.com>
> wrote:
>>
>> Do you have same seed node specified in cass-analysis-1 as cass-1,2,3?
>> I am thinking that changing the seed node in cass-analysis-2 and
>> following the directions in
>> http://wiki.apache.org/cassandra/FAQ#schema_disagreement might solve
>> the problem. Somone please correct me.
>>
>> On Thu, Oct 13, 2011 at 12:05 AM, Eric Czech <eric@nextbigsound.com>
>> wrote:
>> > I don't think that's what I'm after here since the unwanted nodes were
>> > originally assimilated into the cluster with the same initial_token
>> > values
>> > as other nodes that were already in the cluster (that have, and still =
do
>> > have, useful data). =A0I know this is an awkward situation so I'll try=
 to
>> > depict it in a simpler way:
>> > Let's say I have a simplified version of our production cluster that
>> > looks
>> > like this -
>> > cass-1 =A0 token =3D A
>> > cass-2 =A0 token =3D B
>> > cass-3 =A0 token =3D C
>> > Then I tried to create a second cluster that looks like this -
>> > cass-analysis-1 =A0 token =3D A =A0(and contains same data as cass-1)
>> > cass-analysis-2 =A0 token =3D B =A0(and contains same data as cass-2)
>> > cass-analysis-3 =A0 token =3D C =A0(and contains same data as cass-3)
>> > But after starting the second cluster, things got crossed up between t=
he
>> > clusters and here's what the original cluster now looks like -
>> > cass-1 =A0 token =3D A =A0 (has data and schema)
>> > cass-2 =A0 token =3D B =A0 (has data and schema)
>> > cass-3 =A0 token =3D C =A0 (had data and schema)
>> > cass-analysis-1=A0 =A0token =3D A =A0(has *no* data and is not part of=
 the ring,
>> > but
>> > is trying to be included in cluster schema)
>> > A simplified version of "describe cluster" =A0for the original cluster=
 now
>> > shows:
>> > Cluster Information:
>> > =A0 =A0Schema versions:
>> > SCHEMA-UUID-1: [cass-1, cass-2, cass-3]
>> > SCHEMA-UUID-2: [cass-analysis-1]
>> > But the simplified ring looks like this (has only 3 nodes instead of 4=
):
>> > Host =A0 =A0 =A0 Owns =A0 =A0 Token
>> > cass-1 =A0 =A0 33% =A0 =A0 =A0 A
>> > cass-2 =A0 =A0 33% =A0 =A0 =A0 B
>> > cass-3 =A0 =A0 33% =A0 =A0 =A0 C
>> > The original cluster is still working correctly but all live schema
>> > updates
>> > are failing because of the inconsistent schema versions introduced by
>> > the
>> > unwanted node.
>> > From my perspective, a simple fix seems to be for cassandra to exclude
>> > nodes
>> > that aren't part of the ring from the schema consistency requirements.
>> > =A0Any
>> > reason that wouldn't work?
>> > And aside from a possible code patch, any recommendations as to how I
>> > can
>> > best fix this given the current 8.4 release?
>> >
>> > On Thu, Oct 13, 2011 at 12:14 AM, Jonathan Ellis <jbellis@gmail.com>
>> > wrote:
>> >>
>> >> Does nodetool removetoken not work?
>> >>
>> >> On Thu, Oct 13, 2011 at 12:59 AM, Eric Czech <eric@nextbigsound.com>
>> >> wrote:
>> >> > Not sure if anyone has seen this before but it's really killing me
>> >> > right
>> >> > now. =A0Perhaps that was too long of a description of the issue so
>> >> > here's
>> >> > a
>> >> > more succinct question -- How do I remove nodes associated with a
>> >> > cluster
>> >> > that contain no data and have no reason to be associated with the
>> >> > cluster
>> >> > whatsoever?
>> >> > My last resort here is to stop cassandra (after recording all token=
s
>> >> > for
>> >> > each node),=A0set the initial token for each node in the cluster in
>> >> > cassandra.yaml, manually=A0delete the LocationInfo* sstables in the
>> >> > system
>> >> > keyspace, and then restart. =A0I'm hoping there's a simpler, less
>> >> > seemingly
>> >> > risky way to do this so please, please let me know if that's true!
>> >> > Thanks again.
>> >> > - Eric
>> >> > On Tue, Oct 11, 2011 at 11:55 AM, Eric Czech <eric@nextbigsound.com=
>
>> >> > wrote:
>> >> >>
>> >> >> Hi, I'm having what I think is a fairly uncommon schema issue --
>> >> >> My situation is that I had a cluster with 10 nodes and a consisten=
t
>> >> >> schema. =A0Then, in an experiment to setup a second cluster with t=
he
>> >> >> same
>> >> >> information (by copying the raw sstables), I left the LocationInfo=
*
>> >> >> sstables
>> >> >> in the system keyspace in the new cluster and after starting the
>> >> >> second
>> >> >> cluster, I realized that the two clusters were discovering each
>> >> >> other
>> >> >> when
>> >> >> they shouldn't have been. =A0Since then, I changed the cluster nam=
e
>> >> >> for
>> >> >> the
>> >> >> second cluster and made sure to delete the LocationInfo* sstables
>> >> >> before
>> >> >> starting it and the two clusters are now operating independent of
>> >> >> one
>> >> >> another for the most part. =A0The only remaining connection betwee=
n
>> >> >> the
>> >> >> two
>> >> >> seems to be that the first cluster is still maintaining references
>> >> >> to
>> >> >> nodes
>> >> >> in the second cluster in the schema versions despite those nodes n=
ot
>> >> >> actually being part of the ring.
>> >> >> Here's what my "describe cluster" looks like on the original
>> >> >> cluster:
>> >> >> Cluster Information:
>> >> >> =A0 =A0Snitch: org.apache.cassandra.locator.SimpleSnitch
>> >> >> =A0 =A0Partitioner: org.apache.cassandra.dht.RandomPartitioner
>> >> >> =A0 =A0Schema versions:
>> >> >> 48971cb0-e9ff-11e0-0000-eb9eab7d90bf: [<INTENTIONAL_IP1>,
>> >> >> <INTENTIONAL_IP2>, ..., <INTENTIONAL_IP10>]
>> >> >> 848bcfc0-eddf-11e0-0000-8a3bb58f08ff: [<NOT_INTENTIONAL_IP1>,
>> >> >> <NOT_INTENTIONAL_IP2>]
>> >> >> The second cluster, however, contains no schema versions involving
>> >> >> nodes
>> >> >> from the first cluster.
>> >> >> My question then is, how can I remove those schema versions from t=
he
>> >> >> original cluster that are associated with the unwanted nodes from
>> >> >> the
>> >> >> second
>> >> >> cluster? =A0Is there any way to remove or evict an IP from a clust=
er
>> >> >> instead
>> >> >> of just a token?
>> >> >> Thanks in advance!
>> >> >> - Eric
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>
>