cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "B. Todd Burruss" <bburr...@real.com>
Subject Re: ring state out of sync in build 883477
Date Tue, 24 Nov 2009 17:39:40 GMT
grt thx.

on another note, is there a way to know that a node has fully
bootstrapped or resync'ed after a restart?  meaning it has its slice of
the ring, the data replicated from other nodes, etc?  

i've glanced thru the JMX properties but didn't see anything.

thx!

grt work


On Tue, 2009-11-24 at 11:21 -0600, Jonathan Ellis wrote:
> Looks like this is another symptom of
> https://issues.apache.org/jira/browse/CASSANDRA-150, which is on track
> to be fixed soon
> 
> On Tue, Nov 24, 2009 at 11:19 AM, B. Todd Burruss <bburruss@real.com> wrote:
> > they all were restarted at various times.
> >
> > for vmguest85 the other three are seed nodes.
> >
> >
> > On Mon, 2009-11-23 at 19:21 -0600, Jonathan Ellis wrote:
> >> So vmquest85 was restarted, but gen-app02 hasn't told it that there
> >> are 2 other nodes that are down?
> >>
> >> Which one is the seed node?
> >>
> >> On Mon, Nov 23, 2009 at 6:38 PM, B. Todd Burruss <bburruss@real.com> wrote:
> >> > i'm observing the following on a cluster that started with 4 nodes.  i
have
> >> > been killing and restarting the various nodes as i test cassandra and now
> >> > i'm seeing a lot of NotFoundException exceptions in the client because
what
> >> > i believe is ring state out of sync between the two nodes that are still
up
> >> > and available.  The first ring state shown below reflects the current state
> >> > of the cluster.  Also I have seen similar issues when one of the nodes
> >> > thinks another node is still available when in fact it has been killed.
 it
> >> > seems to be related to bringing up, killing nodes too fast and not letting
> >> > them figure out when a node is "dead".  in this case i see TimedOutException
> >> > related to NIO SocketChannel class.
> >> >
> >> > thx!
> >> >
> >> > [cassandra.883477]$ bin/nodeprobe -host gen-app02.dev.real.com -port 8080
> >> > ring
> >> > Address       Status     Load
> >> > Range                                      Ring
> >> >
> >> > 144038903974614862325597275257769797985
> >> > 172.27.128.186Down       22.17 MB
> >> > 31124469348629903091013930339840898757     |<--|
> >> > 172.27.128.23 Down       22.17 MB
> >> > 64378740291415296162944450043143967518     |   |
> >> > 172.27.128.22 Up         22.17 MB
> >> > 121134220722269938669001112695509564769    |   |
> >> > 172.27.128.185Up         14.69 MB
> >> > 144038903974614862325597275257769797985    |-->|
> >> >
> >> > [cassandra.883477]$ bin/nodeprobe -host vmguest85.prognet.com -port 8080
> >> > ring
> >> > Address       Status     Load
> >> > Range                                      Ring
> >> >
> >> > 144038903974614862325597275257769797985
> >> > 172.27.128.22 Up         22.17 MB
> >> > 121134220722269938669001112695509564769    |<--|
> >> > 172.27.128.185Up         14.69 MB
> >> > 144038903974614862325597275257769797985    |-->|
> >> > [cassandra.883477]$
> >> >
> >> >
> >> >
> >
> >
> >



Mime
View raw message