incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <ran...@gmail.com>
Subject RuntimeException: Cannot service reads while bootstrapping!
Date Wed, 04 Aug 2010 10:52:27 GMT
Is this a known issue?
Running 0.6.2 I moved a node to different token and eventually saw errors in
the log.


ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
java.lang.RuntimeException: Cannot service reads while bootstrapping!
        at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700 CassandraDaemon.java
(line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
java.lang.RuntimeException: Cannot service reads while bootstrapping!
        at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
... many more of those and then...
 INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
StorageService.java (line 181) Bootstrap/move completed! Now serving reads.


The move ended up "ok" but during the operation the log was filled with
those errors and at the end of it the ring state was inconsistent.
If I ask the moved node where it is in the ring it tells me something but
other nodes tell something else...

(ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
192.168.254.58 -p 9004 ring
Address       Status     Load          Range
     Ring

170141183460469231731687303715884105727
192.168.252.88Up         5.7 GB
 14131484407726020523932116250949797205     |<--|
192.168.252.124Up         2.44 GB
56713727820156410577229101238628035242     |   ^
192.168.254.58Up         8.13 GB       *
113427455640312821154458202477256070484*    v   |
192.168.254.57Up         6.52 GB
113427455640312821154458202477256070485    |   ^
192.168.252.125Up         6.52 GB
141784319550391026443072753096570088105    v   |
192.168.254.59Up         1.63 GB
170141183460469231731687303715884105727    |-->|
(ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
192.168.252.124 -p 9004 ring
Address       Status     Load          Range
     Ring

170141183460469231731687303715884105727
192.168.252.88Up         5.7 GB
 14131484407726020523932116250949797205     |<--|
192.168.252.124Up         2.46 GB
56713727820156410577229101238628035242     |   ^
192.168.254.57Up         6.52 GB
113427455640312821154458202477256070485    v   |
192.168.252.125Up         6.52 GB
141784319550391026443072753096570088105    |   ^
192.168.254.58Up         1.63 GB       *
141784319550391026443072753096570088106*    v   |
192.168.254.59Up         1.63 GB
170141183460469231731687303715884105727    |-->|

Restarting the moved node fixes the ring view by other hosts.

Mime
View raw message