incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Re: Schema versions reflect schemas on unwanted nodes
Date Thu, 13 Oct 2011 14:01:08 GMT
Do you have same seed node specified in cass-analysis-1 as cass-1,2,3?
I am thinking that changing the seed node in cass-analysis-2 and
following the directions in
http://wiki.apache.org/cassandra/FAQ#schema_disagreement might solve
the problem. Somone please correct me.

On Thu, Oct 13, 2011 at 12:05 AM, Eric Czech <eric@nextbigsound.com> wrote:
> I don't think that's what I'm after here since the unwanted nodes were
> originally assimilated into the cluster with the same initial_token values
> as other nodes that were already in the cluster (that have, and still do
> have, useful data).  I know this is an awkward situation so I'll try to
> depict it in a simpler way:
> Let's say I have a simplified version of our production cluster that looks
> like this -
> cass-1   token = A
> cass-2   token = B
> cass-3   token = C
> Then I tried to create a second cluster that looks like this -
> cass-analysis-1   token = A  (and contains same data as cass-1)
> cass-analysis-2   token = B  (and contains same data as cass-2)
> cass-analysis-3   token = C  (and contains same data as cass-3)
> But after starting the second cluster, things got crossed up between the
> clusters and here's what the original cluster now looks like -
> cass-1   token = A   (has data and schema)
> cass-2   token = B   (has data and schema)
> cass-3   token = C   (had data and schema)
> cass-analysis-1   token = A  (has *no* data and is not part of the ring, but
> is trying to be included in cluster schema)
> A simplified version of "describe cluster"  for the original cluster now
> shows:
> Cluster Information:
>    Schema versions:
> SCHEMA-UUID-1: [cass-1, cass-2, cass-3]
> SCHEMA-UUID-2: [cass-analysis-1]
> But the simplified ring looks like this (has only 3 nodes instead of 4):
> Host       Owns     Token
> cass-1     33%       A
> cass-2     33%       B
> cass-3     33%       C
> The original cluster is still working correctly but all live schema updates
> are failing because of the inconsistent schema versions introduced by the
> unwanted node.
> From my perspective, a simple fix seems to be for cassandra to exclude nodes
> that aren't part of the ring from the schema consistency requirements.  Any
> reason that wouldn't work?
> And aside from a possible code patch, any recommendations as to how I can
> best fix this given the current 8.4 release?
>
> On Thu, Oct 13, 2011 at 12:14 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> Does nodetool removetoken not work?
>>
>> On Thu, Oct 13, 2011 at 12:59 AM, Eric Czech <eric@nextbigsound.com>
>> wrote:
>> > Not sure if anyone has seen this before but it's really killing me right
>> > now.  Perhaps that was too long of a description of the issue so here's
>> > a
>> > more succinct question -- How do I remove nodes associated with a
>> > cluster
>> > that contain no data and have no reason to be associated with the
>> > cluster
>> > whatsoever?
>> > My last resort here is to stop cassandra (after recording all tokens for
>> > each node), set the initial token for each node in the cluster in
>> > cassandra.yaml, manually delete the LocationInfo* sstables in the system
>> > keyspace, and then restart.  I'm hoping there's a simpler, less
>> > seemingly
>> > risky way to do this so please, please let me know if that's true!
>> > Thanks again.
>> > - Eric
>> > On Tue, Oct 11, 2011 at 11:55 AM, Eric Czech <eric@nextbigsound.com>
>> > wrote:
>> >>
>> >> Hi, I'm having what I think is a fairly uncommon schema issue --
>> >> My situation is that I had a cluster with 10 nodes and a consistent
>> >> schema.  Then, in an experiment to setup a second cluster with the same
>> >> information (by copying the raw sstables), I left the LocationInfo*
>> >> sstables
>> >> in the system keyspace in the new cluster and after starting the second
>> >> cluster, I realized that the two clusters were discovering each other
>> >> when
>> >> they shouldn't have been.  Since then, I changed the cluster name for
>> >> the
>> >> second cluster and made sure to delete the LocationInfo* sstables
>> >> before
>> >> starting it and the two clusters are now operating independent of one
>> >> another for the most part.  The only remaining connection between the
>> >> two
>> >> seems to be that the first cluster is still maintaining references to
>> >> nodes
>> >> in the second cluster in the schema versions despite those nodes not
>> >> actually being part of the ring.
>> >> Here's what my "describe cluster" looks like on the original cluster:
>> >> Cluster Information:
>> >>    Snitch: org.apache.cassandra.locator.SimpleSnitch
>> >>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>> >>    Schema versions:
>> >> 48971cb0-e9ff-11e0-0000-eb9eab7d90bf: [<INTENTIONAL_IP1>,
>> >> <INTENTIONAL_IP2>, ..., <INTENTIONAL_IP10>]
>> >> 848bcfc0-eddf-11e0-0000-8a3bb58f08ff: [<NOT_INTENTIONAL_IP1>,
>> >> <NOT_INTENTIONAL_IP2>]
>> >> The second cluster, however, contains no schema versions involving
>> >> nodes
>> >> from the first cluster.
>> >> My question then is, how can I remove those schema versions from the
>> >> original cluster that are associated with the unwanted nodes from the
>> >> second
>> >> cluster?  Is there any way to remove or evict an IP from a cluster
>> >> instead
>> >> of just a token?
>> >> Thanks in advance!
>> >> - Eric
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>

Mime
View raw message