Don't point clients at nodes that aren't part of the ring. Cassandra
rejecting requests when you do is a "feature."
On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <rantav@gmail.com> wrote:
> Is this a known issue?
> Running 0.6.2 I moved a node to different token and eventually saw errors in
> the log.
>
> ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
> DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
> java.lang.RuntimeException: Cannot service reads while bootstrapping!
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700 CassandraDaemon.java
> (line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
> java.lang.RuntimeException: Cannot service reads while bootstrapping!
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ... many more of those and then...
> INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
> StorageService.java (line 181) Bootstrap/move completed! Now serving reads.
>
> The move ended up "ok" but during the operation the log was filled with
> those errors and at the end of it the ring state was inconsistent.
> If I ask the moved node where it is in the ring it tells me something but
> other nodes tell something else...
> (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
> 192.168.254.58 -p 9004 ring
> Address Status Load Range
> Ring
>
> 170141183460469231731687303715884105727
> 192.168.252.88Up 5.7 GB
> 14131484407726020523932116250949797205 |<--|
> 192.168.252.124Up 2.44 GB
> 56713727820156410577229101238628035242 | ^
> 192.168.254.58Up 8.13 GB
> 113427455640312821154458202477256070484 v |
> 192.168.254.57Up 6.52 GB
> 113427455640312821154458202477256070485 | ^
> 192.168.252.125Up 6.52 GB
> 141784319550391026443072753096570088105 v |
> 192.168.254.59Up 1.63 GB
> 170141183460469231731687303715884105727 |-->|
> (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
> -h 192.168.252.124 -p 9004 ring
> Address Status Load Range
> Ring
>
> 170141183460469231731687303715884105727
> 192.168.252.88Up 5.7 GB
> 14131484407726020523932116250949797205 |<--|
> 192.168.252.124Up 2.46 GB
> 56713727820156410577229101238628035242 | ^
> 192.168.254.57Up 6.52 GB
> 113427455640312821154458202477256070485 v |
> 192.168.252.125Up 6.52 GB
> 141784319550391026443072753096570088105 | ^
> 192.168.254.58Up 1.63 GB
> 141784319550391026443072753096570088106 v |
> 192.168.254.59Up 1.63 GB
> 170141183460469231731687303715884105727 |-->|
> Restarting the moved node fixes the ring view by other hosts.
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
|