reported https://issues.apache.org/jira/browse/CASSANDRA-1359

On Wed, Aug 4, 2010 at 9:51 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
That's definitely a bug.

On Wed, Aug 4, 2010 at 2:13 PM, Ran Tavory <rantav@gmail.com> wrote:
> ok, so I don't send writes to bootstrapping or decommissioned nodes, that's
> cool, but what about the inconsistent ring view after nodetool move, isn't
> this strange?
> After the move, the moved node has the correct view of the ring but all
> other nodes have the old view. I waited a few minutes after the log said
> that Bootstrap/move completed! Now serving reads but this didn't help, view
> was still inconsistent. Only restarting the moved node helped other nodes
> realize the change.
> On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> Don't point clients at nodes that aren't part of the ring.  Cassandra
>> rejecting requests when you do is a "feature."
>>
>> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <rantav@gmail.com> wrote:
>> > Is this a known issue?
>> > Running 0.6.2 I moved a node to different token and eventually saw
>> > errors in
>> > the log.
>> >
>> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
>> > DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
>> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
>> >         at
>> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
>> >         at
>> >
>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >         at java.lang.Thread.run(Thread.java:619)
>> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700
>> > CassandraDaemon.java
>> > (line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
>> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
>> >         at
>> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
>> >         at
>> >
>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >         at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >         at java.lang.Thread.run(Thread.java:619)
>> > ... many more of those and then...
>> >  INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
>> > StorageService.java (line 181) Bootstrap/move completed! Now serving
>> > reads.
>> >
>> > The move ended up "ok" but during the operation the log was filled with
>> > those errors and at the end of it the ring state was inconsistent.
>> > If I ask the moved node where it is in the ring it tells me something
>> > but
>> > other nodes tell something else...
>> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
>> > 192.168.254.58 -p 9004 ring
>> > Address       Status     Load          Range
>> >      Ring
>> >
>> > 170141183460469231731687303715884105727
>> > 192.168.252.88Up         5.7 GB
>> >  14131484407726020523932116250949797205     |<--|
>> > 192.168.252.124Up         2.44 GB
>> > 56713727820156410577229101238628035242     |   ^
>> > 192.168.254.58Up         8.13 GB
>> > 113427455640312821154458202477256070484    v   |
>> > 192.168.254.57Up         6.52 GB
>> > 113427455640312821154458202477256070485    |   ^
>> > 192.168.252.125Up         6.52 GB
>> > 141784319550391026443072753096570088105    v   |
>> > 192.168.254.59Up         1.63 GB
>> > 170141183460469231731687303715884105727    |-->|
>> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
>> > -h 192.168.252.124 -p 9004 ring
>> > Address       Status     Load          Range
>> >      Ring
>> >
>> > 170141183460469231731687303715884105727
>> > 192.168.252.88Up         5.7 GB
>> >  14131484407726020523932116250949797205     |<--|
>> > 192.168.252.124Up         2.46 GB
>> > 56713727820156410577229101238628035242     |   ^
>> > 192.168.254.57Up         6.52 GB
>> > 113427455640312821154458202477256070485    v   |
>> > 192.168.252.125Up         6.52 GB
>> > 141784319550391026443072753096570088105    |   ^
>> > 192.168.254.58Up         1.63 GB
>> > 141784319550391026443072753096570088106    v   |
>> > 192.168.254.59Up         1.63 GB
>> > 170141183460469231731687303715884105727    |-->|
>> > Restarting the moved node fixes the ring view by other hosts.
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com