cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "B. Todd Burruss" <bburr...@real.com>
Subject Re: ring state out of sync in build 883477
Date Tue, 24 Nov 2009 17:19:14 GMT
they all were restarted at various times.

for vmguest85 the other three are seed nodes.


On Mon, 2009-11-23 at 19:21 -0600, Jonathan Ellis wrote:
> So vmquest85 was restarted, but gen-app02 hasn't told it that there
> are 2 other nodes that are down?
> 
> Which one is the seed node?
> 
> On Mon, Nov 23, 2009 at 6:38 PM, B. Todd Burruss <bburruss@real.com> wrote:
> > i'm observing the following on a cluster that started with 4 nodes.  i have
> > been killing and restarting the various nodes as i test cassandra and now
> > i'm seeing a lot of NotFoundException exceptions in the client because what
> > i believe is ring state out of sync between the two nodes that are still up
> > and available.  The first ring state shown below reflects the current state
> > of the cluster.  Also I have seen similar issues when one of the nodes
> > thinks another node is still available when in fact it has been killed.  it
> > seems to be related to bringing up, killing nodes too fast and not letting
> > them figure out when a node is "dead".  in this case i see TimedOutException
> > related to NIO SocketChannel class.
> >
> > thx!
> >
> > [cassandra.883477]$ bin/nodeprobe -host gen-app02.dev.real.com -port 8080
> > ring
> > Address       Status     Load
> > Range                                      Ring
> >
> > 144038903974614862325597275257769797985
> > 172.27.128.186Down       22.17 MB
> > 31124469348629903091013930339840898757     |<--|
> > 172.27.128.23 Down       22.17 MB
> > 64378740291415296162944450043143967518     |   |
> > 172.27.128.22 Up         22.17 MB
> > 121134220722269938669001112695509564769    |   |
> > 172.27.128.185Up         14.69 MB
> > 144038903974614862325597275257769797985    |-->|
> >
> > [cassandra.883477]$ bin/nodeprobe -host vmguest85.prognet.com -port 8080
> > ring
> > Address       Status     Load
> > Range                                      Ring
> >
> > 144038903974614862325597275257769797985
> > 172.27.128.22 Up         22.17 MB
> > 121134220722269938669001112695509564769    |<--|
> > 172.27.128.185Up         14.69 MB
> > 144038903974614862325597275257769797985    |-->|
> > [cassandra.883477]$
> >
> >
> >



Mime
View raw message