incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: RuntimeException: Cannot service reads while bootstrapping!
Date Wed, 04 Aug 2010 12:24:45 GMT
Don't point clients at nodes that aren't part of the ring.  Cassandra
rejecting requests when you do is a "feature."

On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <rantav@gmail.com> wrote:
> Is this a known issue?
> Running 0.6.2 I moved a node to different token and eventually saw errors in
> the log.
>
> ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
> DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
> java.lang.RuntimeException: Cannot service reads while bootstrapping!
>         at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700 CassandraDaemon.java
> (line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
> java.lang.RuntimeException: Cannot service reads while bootstrapping!
>         at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> ... many more of those and then...
>  INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
> StorageService.java (line 181) Bootstrap/move completed! Now serving reads.
>
> The move ended up "ok" but during the operation the log was filled with
> those errors and at the end of it the ring state was inconsistent.
> If I ask the moved node where it is in the ring it tells me something but
> other nodes tell something else...
> (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
> 192.168.254.58 -p 9004 ring
> Address       Status     Load          Range
>      Ring
>
> 170141183460469231731687303715884105727
> 192.168.252.88Up         5.7 GB
>  14131484407726020523932116250949797205     |<--|
> 192.168.252.124Up         2.44 GB
> 56713727820156410577229101238628035242     |   ^
> 192.168.254.58Up         8.13 GB
> 113427455640312821154458202477256070484    v   |
> 192.168.254.57Up         6.52 GB
> 113427455640312821154458202477256070485    |   ^
> 192.168.252.125Up         6.52 GB
> 141784319550391026443072753096570088105    v   |
> 192.168.254.59Up         1.63 GB
> 170141183460469231731687303715884105727    |-->|
> (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
> -h 192.168.252.124 -p 9004 ring
> Address       Status     Load          Range
>      Ring
>
> 170141183460469231731687303715884105727
> 192.168.252.88Up         5.7 GB
>  14131484407726020523932116250949797205     |<--|
> 192.168.252.124Up         2.46 GB
> 56713727820156410577229101238628035242     |   ^
> 192.168.254.57Up         6.52 GB
> 113427455640312821154458202477256070485    v   |
> 192.168.252.125Up         6.52 GB
> 141784319550391026443072753096570088105    |   ^
> 192.168.254.58Up         1.63 GB
> 141784319550391026443072753096570088106    v   |
> 192.168.254.59Up         1.63 GB
> 170141183460469231731687303715884105727    |-->|
> Restarting the moved node fixes the ring view by other hosts.
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message