cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: ring state out of sync in build 883477
Date Tue, 24 Nov 2009 17:21:30 GMT
Looks like this is another symptom of
https://issues.apache.org/jira/browse/CASSANDRA-150, which is on track
to be fixed soon

On Tue, Nov 24, 2009 at 11:19 AM, B. Todd Burruss <bburruss@real.com> wrote:
> they all were restarted at various times.
>
> for vmguest85 the other three are seed nodes.
>
>
> On Mon, 2009-11-23 at 19:21 -0600, Jonathan Ellis wrote:
>> So vmquest85 was restarted, but gen-app02 hasn't told it that there
>> are 2 other nodes that are down?
>>
>> Which one is the seed node?
>>
>> On Mon, Nov 23, 2009 at 6:38 PM, B. Todd Burruss <bburruss@real.com> wrote:
>> > i'm observing the following on a cluster that started with 4 nodes.  i have
>> > been killing and restarting the various nodes as i test cassandra and now
>> > i'm seeing a lot of NotFoundException exceptions in the client because what
>> > i believe is ring state out of sync between the two nodes that are still up
>> > and available.  The first ring state shown below reflects the current state
>> > of the cluster.  Also I have seen similar issues when one of the nodes
>> > thinks another node is still available when in fact it has been killed.  it
>> > seems to be related to bringing up, killing nodes too fast and not letting
>> > them figure out when a node is "dead".  in this case i see TimedOutException
>> > related to NIO SocketChannel class.
>> >
>> > thx!
>> >
>> > [cassandra.883477]$ bin/nodeprobe -host gen-app02.dev.real.com -port 8080
>> > ring
>> > Address       Status     Load
>> > Range                                      Ring
>> >
>> > 144038903974614862325597275257769797985
>> > 172.27.128.186Down       22.17 MB
>> > 31124469348629903091013930339840898757     |<--|
>> > 172.27.128.23 Down       22.17 MB
>> > 64378740291415296162944450043143967518     |   |
>> > 172.27.128.22 Up         22.17 MB
>> > 121134220722269938669001112695509564769    |   |
>> > 172.27.128.185Up         14.69 MB
>> > 144038903974614862325597275257769797985    |-->|
>> >
>> > [cassandra.883477]$ bin/nodeprobe -host vmguest85.prognet.com -port 8080
>> > ring
>> > Address       Status     Load
>> > Range                                      Ring
>> >
>> > 144038903974614862325597275257769797985
>> > 172.27.128.22 Up         22.17 MB
>> > 121134220722269938669001112695509564769    |<--|
>> > 172.27.128.185Up         14.69 MB
>> > 144038903974614862325597275257769797985    |-->|
>> > [cassandra.883477]$
>> >
>> >
>> >
>
>
>

Mime
View raw message