cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <ran...@gmail.com>
Subject Re: RuntimeException: Cannot service reads while bootstrapping!
Date Wed, 04 Aug 2010 18:13:28 GMT
ok, so I don't send writes to bootstrapping or decommissioned nodes, that's
cool, but what about the inconsistent ring view after nodetool move, isn't
this strange?
After the move, the moved node has the correct view of the ring but all
other nodes have the old view. I waited a few minutes after the log said
that Bootstrap/move completed! Now serving reads but this didn't help, view
was still inconsistent. Only restarting the moved node helped other nodes
realize the change.

On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> Don't point clients at nodes that aren't part of the ring.  Cassandra
> rejecting requests when you do is a "feature."
>
> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <rantav@gmail.com> wrote:
> > Is this a known issue?
> > Running 0.6.2 I moved a node to different token and eventually saw errors
> in
> > the log.
> >
> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
> > DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
> >         at
> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> >         at
> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >         at java.lang.Thread.run(Thread.java:619)
> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700
> CassandraDaemon.java
> > (line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,main]
> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
> >         at
> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> >         at
> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >         at java.lang.Thread.run(Thread.java:619)
> > ... many more of those and then...
> >  INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
> > StorageService.java (line 181) Bootstrap/move completed! Now serving
> reads.
> >
> > The move ended up "ok" but during the operation the log was filled with
> > those errors and at the end of it the ring state was inconsistent.
> > If I ask the moved node where it is in the ring it tells me something but
> > other nodes tell something else...
> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
> > 192.168.254.58 -p 9004 ring
> > Address       Status     Load          Range
> >      Ring
> >
> > 170141183460469231731687303715884105727
> > 192.168.252.88Up         5.7 GB
> >  14131484407726020523932116250949797205     |<--|
> > 192.168.252.124Up         2.44 GB
> > 56713727820156410577229101238628035242     |   ^
> > 192.168.254.58Up         8.13 GB
> > 113427455640312821154458202477256070484    v   |
> > 192.168.254.57Up         6.52 GB
> > 113427455640312821154458202477256070485    |   ^
> > 192.168.252.125Up         6.52 GB
> > 141784319550391026443072753096570088105    v   |
> > 192.168.254.59Up         1.63 GB
> > 170141183460469231731687303715884105727    |-->|
> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
> > -h 192.168.252.124 -p 9004 ring
> > Address       Status     Load          Range
> >      Ring
> >
> > 170141183460469231731687303715884105727
> > 192.168.252.88Up         5.7 GB
> >  14131484407726020523932116250949797205     |<--|
> > 192.168.252.124Up         2.46 GB
> > 56713727820156410577229101238628035242     |   ^
> > 192.168.254.57Up         6.52 GB
> > 113427455640312821154458202477256070485    v   |
> > 192.168.252.125Up         6.52 GB
> > 141784319550391026443072753096570088105    |   ^
> > 192.168.254.58Up         1.63 GB
> > 141784319550391026443072753096570088106    v   |
> > 192.168.254.59Up         1.63 GB
> > 170141183460469231731687303715884105727    |-->|
> > Restarting the moved node fixes the ring view by other hosts.
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Mime
View raw message