cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Czech <e...@nextbigsound.com>
Subject Re: Schema versions reflect schemas on unwanted nodes
Date Thu, 13 Oct 2011 07:05:58 GMT
I don't think that's what I'm after here since the unwanted nodes were
originally assimilated into the cluster with the same initial_token values
as other nodes that were already in the cluster (that have, and still do
have, useful data).  I know this is an awkward situation so I'll try to
depict it in a simpler way:

Let's say I have a simplified version of our production cluster that looks
like this -

cass-1   token = A
cass-2   token = B
cass-3   token = C

Then I tried to create a second cluster that looks like this -

cass-analysis-1   token = A  (and contains same data as cass-1)
cass-analysis-2   token = B  (and contains same data as cass-2)
cass-analysis-3   token = C  (and contains same data as cass-3)

But after starting the second cluster, things got crossed up between the
clusters and here's what the original cluster now looks like -

cass-1   token = A   (has data and schema)
cass-2   token = B   (has data and schema)
cass-3   token = C   (had data and schema)
cass-analysis-1   token = A  (has *no* data and is not part of the ring, but
is trying to be included in cluster schema)

A simplified version of "describe cluster"  for the original cluster now
shows:

Cluster Information:
   Schema versions:
 SCHEMA-UUID-1: [cass-1, cass-2, cass-3]
SCHEMA-UUID-2: [cass-analysis-1]

But the simplified ring looks like this (has only 3 nodes instead of 4):

Host       Owns     Token
cass-1     33%       A
cass-2     33%       B
cass-3     33%       C

The original cluster is still working correctly but all live schema updates
are failing because of the inconsistent schema versions introduced by the
unwanted node.

>From my perspective, a simple fix seems to be for cassandra to exclude nodes
that aren't part of the ring from the schema consistency requirements.  Any
reason that wouldn't work?

And aside from a possible code patch, any recommendations as to how I can
best fix this given the current 8.4 release?


On Thu, Oct 13, 2011 at 12:14 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

> Does nodetool removetoken not work?
>
> On Thu, Oct 13, 2011 at 12:59 AM, Eric Czech <eric@nextbigsound.com>
> wrote:
> > Not sure if anyone has seen this before but it's really killing me right
> > now.  Perhaps that was too long of a description of the issue so here's a
> > more succinct question -- How do I remove nodes associated with a cluster
> > that contain no data and have no reason to be associated with the cluster
> > whatsoever?
> > My last resort here is to stop cassandra (after recording all tokens for
> > each node), set the initial token for each node in the cluster in
> > cassandra.yaml, manually delete the LocationInfo* sstables in the system
> > keyspace, and then restart.  I'm hoping there's a simpler, less seemingly
> > risky way to do this so please, please let me know if that's true!
> > Thanks again.
> > - Eric
> > On Tue, Oct 11, 2011 at 11:55 AM, Eric Czech <eric@nextbigsound.com>
> wrote:
> >>
> >> Hi, I'm having what I think is a fairly uncommon schema issue --
> >> My situation is that I had a cluster with 10 nodes and a consistent
> >> schema.  Then, in an experiment to setup a second cluster with the same
> >> information (by copying the raw sstables), I left the LocationInfo*
> sstables
> >> in the system keyspace in the new cluster and after starting the second
> >> cluster, I realized that the two clusters were discovering each other
> when
> >> they shouldn't have been.  Since then, I changed the cluster name for
> the
> >> second cluster and made sure to delete the LocationInfo* sstables before
> >> starting it and the two clusters are now operating independent of one
> >> another for the most part.  The only remaining connection between the
> two
> >> seems to be that the first cluster is still maintaining references to
> nodes
> >> in the second cluster in the schema versions despite those nodes not
> >> actually being part of the ring.
> >> Here's what my "describe cluster" looks like on the original cluster:
> >> Cluster Information:
> >>    Snitch: org.apache.cassandra.locator.SimpleSnitch
> >>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
> >>    Schema versions:
> >> 48971cb0-e9ff-11e0-0000-eb9eab7d90bf: [<INTENTIONAL_IP1>,
> >> <INTENTIONAL_IP2>, ..., <INTENTIONAL_IP10>]
> >> 848bcfc0-eddf-11e0-0000-8a3bb58f08ff: [<NOT_INTENTIONAL_IP1>,
> >> <NOT_INTENTIONAL_IP2>]
> >> The second cluster, however, contains no schema versions involving nodes
> >> from the first cluster.
> >> My question then is, how can I remove those schema versions from the
> >> original cluster that are associated with the unwanted nodes from the
> second
> >> cluster?  Is there any way to remove or evict an IP from a cluster
> instead
> >> of just a token?
> >> Thanks in advance!
> >> - Eric
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

Mime
View raw message