incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <ran...@gmail.com>
Subject Re: RuntimeException: Cannot service reads while bootstrapping!
Date Wed, 04 Aug 2010 19:50:17 GMT
reported https://issues.apache.org/jira/browse/CASSANDRA-1359

On Wed, Aug 4, 2010 at 9:51 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> That's definitely a bug.
>
> On Wed, Aug 4, 2010 at 2:13 PM, Ran Tavory <rantav@gmail.com> wrote:
> > ok, so I don't send writes to bootstrapping or decommissioned nodes,
> that's
> > cool, but what about the inconsistent ring view after nodetool move,
> isn't
> > this strange?
> > After the move, the moved node has the correct view of the ring but all
> > other nodes have the old view. I waited a few minutes after the log said
> > that Bootstrap/move completed! Now serving reads but this didn't help,
> view
> > was still inconsistent. Only restarting the moved node helped other nodes
> > realize the change.
> > On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis <jbellis@gmail.com>
> wrote:
> >>
> >> Don't point clients at nodes that aren't part of the ring.  Cassandra
> >> rejecting requests when you do is a "feature."
> >>
> >> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <rantav@gmail.com> wrote:
> >> > Is this a known issue?
> >> > Running 0.6.2 I moved a node to different token and eventually saw
> >> > errors in
> >> > the log.
> >> >
> >> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
> >> > DebuggableThreadPoolExecutor.java (line 101) Error in
> ThreadPoolExecutor
> >> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
> >> >         at
> >> >
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> >> >         at
> >> >
> >> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> >> >         at
> >> >
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >         at
> >> >
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >         at java.lang.Thread.run(Thread.java:619)
> >> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700
> >> > CassandraDaemon.java
> >> > (line 82) Fatal exception in thread
> Thread[ROW-READ-STAGE:116805,5,main]
> >> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
> >> >         at
> >> >
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66)
> >> >         at
> >> >
> >> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> >> >         at
> >> >
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >         at
> >> >
> >> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >         at java.lang.Thread.run(Thread.java:619)
> >> > ... many more of those and then...
> >> >  INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
> >> > StorageService.java (line 181) Bootstrap/move completed! Now serving
> >> > reads.
> >> >
> >> > The move ended up "ok" but during the operation the log was filled
> with
> >> > those errors and at the end of it the ring state was inconsistent.
> >> > If I ask the moved node where it is in the ring it tells me something
> >> > but
> >> > other nodes tell something else...
> >> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
> >> > 192.168.254.58 -p 9004 ring
> >> > Address       Status     Load          Range
> >> >      Ring
> >> >
> >> > 170141183460469231731687303715884105727
> >> > 192.168.252.88Up         5.7 GB
> >> >  14131484407726020523932116250949797205     |<--|
> >> > 192.168.252.124Up         2.44 GB
> >> > 56713727820156410577229101238628035242     |   ^
> >> > 192.168.254.58Up         8.13 GB
> >> > 113427455640312821154458202477256070484    v   |
> >> > 192.168.254.57Up         6.52 GB
> >> > 113427455640312821154458202477256070485    |   ^
> >> > 192.168.252.125Up         6.52 GB
> >> > 141784319550391026443072753096570088105    v   |
> >> > 192.168.254.59Up         1.63 GB
> >> > 170141183460469231731687303715884105727    |-->|
> >> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
> >> > -h 192.168.252.124 -p 9004 ring
> >> > Address       Status     Load          Range
> >> >      Ring
> >> >
> >> > 170141183460469231731687303715884105727
> >> > 192.168.252.88Up         5.7 GB
> >> >  14131484407726020523932116250949797205     |<--|
> >> > 192.168.252.124Up         2.46 GB
> >> > 56713727820156410577229101238628035242     |   ^
> >> > 192.168.254.57Up         6.52 GB
> >> > 113427455640312821154458202477256070485    v   |
> >> > 192.168.252.125Up         6.52 GB
> >> > 141784319550391026443072753096570088105    |   ^
> >> > 192.168.254.58Up         1.63 GB
> >> > 141784319550391026443072753096570088106    v   |
> >> > 192.168.254.59Up         1.63 GB
> >> > 170141183460469231731687303715884105727    |-->|
> >> > Restarting the moved node fixes the ring view by other hosts.
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of Riptano, the source for professional Cassandra support
> >> http://riptano.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Mime
View raw message