incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Schema consistently not propagating to a node.
Date Fri, 14 Sep 2012 08:38:52 GMT
Out of interest, how out of sync where they ?

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 14/09/2012, at 6:53 AM, Ben Frank <ben@airlust.com> wrote:

> Hi Sergey,
>    That was exactly it, thank you!
> 
> -Ben
> 
> On Thu, Sep 13, 2012 at 12:07 AM, Sergey Tryuber <stryuber@gmail.com> wrote:
> Hi Ben,
> 
> We've seen similar behavior once a time. The cause was unsynchronized time on the nodes.
So I can recommend to check that system time is equal over the nodes (especially on that one
"bad" server).
> 
> 
> On 13 September 2012 05:15, Ben Frank <ben@airlust.com> wrote:
> Hey all,
>    I'm setting up a new ring on 11 machines with cassandra 1.1.5. All seems to install
fine and startup ok (from tarball) but I'm having an issue when updating the schema. There's
one node that just doesn't want to receive the schema change. I've tried blowing away my /var/lib/cassandra/*
directories and trying it a few times with the same results. I also tried the instructions
here: http://wiki.apache.org/cassandra/FAQ#schema_disagreement
> 
> If I update the schema with cassandra-cli I get the following error:
> 
> Waiting for schema agreement...
> The schema has not settled in 10 seconds; further migrations are ill-advised until it
does.
> Versions are 3b172ce8-d8e8-362f-b955-79fe6b8a35e4:[12.19.103.103, 12.19.110.103, 12.19.102.137,
12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105, 172.19.102.135, 12.19.103.105,
12.19.102.133], b83fe28a-2851-34cb-bd8d-7d2621c1d872:[12.19.110.109]
> 
> then if I do a 'describe cluster' I get:
> 
> Cluster Information:
>    Snitch: org.apache.cassandra.locator.SimpleSnitch
>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>    Schema versions: 
> 	3b172ce8-d8e8-362f-b955-79fe6b8a35e4: [12.19.103.103, 12.19.110.103, 12.19.102.137,
12.19.103.111, 12.19.102.131, 12.19.102.129, 12.19.110.105, 12.19.102.135, 12.19.103.105,
12.19.102.133]
> 
> 	a0f80d21-ad98-31df-b6dc-871061b1bcf9: [12.19.110.109]
> 
> if I keep running describe cluster, both the schema UUID's(?) above keep changing every
few seconds. Looking at the logs shows the nodes are continually doing a flush and compact.

> 
> if I start over by blowing away /var/lib/cassandra, and then try the schema update on
the machine which is having trouble receiving it everything works ok and all nodes are in
sync.
> 
> Anyone know what's going on here, or what I should look at to troubleshoot this?
> 
> -Ben
> 
> 


Mime
View raw message