Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 74.125.82.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=Iaj/SbC8/zdXOGTSzikkekm/992iEeh8Y90a0GBTr8eFGcVyr6EczYX8mH3uLnOOKK
         hRgi/EwxqwqD7W97cNHGZRia6GX2DCecxgewdkhBARH5ITcl/om+j2G1G9cFFRhRZJNj
         2iK7JRSlqbg/9Qh5rGUQkbd9Y1QvBWWn93QL0=
MIME-Version: 1.0
In-Reply-To: <AANLkTim_r3cLC8xejBUQ_ia5noOuaPqCXT-kh7WDmMaB@mail.gmail.com>
References: <AANLkTilG3LVMUtATToDCElIxEsXXs3z1gb2y4Nzo8yDQ@mail.gmail.com>
	<AANLkTikNU1YDbeQ4CUZaYzYQTmPGL2v3ckHjEDrcUDAF@mail.gmail.com>
	<AANLkTim_r3cLC8xejBUQ_ia5noOuaPqCXT-kh7WDmMaB@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Wed, 4 Aug 2010 14:51:24 -0400
Message-ID: <AANLkTikW+dbQpugDF2jud7erP8jw7u3cxzJXovjO1r0Y@mail.gmail.com>
Subject: Re: RuntimeException: Cannot service reads while bootstrapping!
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

That's definitely a bug.

On Wed, Aug 4, 2010 at 2:13 PM, Ran Tavory <rantav@gmail.com> wrote:
> ok, so I don't send writes to bootstrapping or=A0decommissioned=A0nodes, =
that's
> cool, but what about the inconsistent ring view after nodetool move, isn'=
t
> this strange?
> After the move, the moved node has the correct view of the ring but all
> other nodes have the old view. I waited a few minutes after the log said
> that=A0Bootstrap/move completed! Now serving reads but this didn't help, =
view
> was still inconsistent. Only=A0restarting=A0the moved node helped other n=
odes
> realize the change.
> On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> Don't point clients at nodes that aren't part of the ring. =A0Cassandra
>> rejecting requests when you do is a "feature."
>>
>> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <rantav@gmail.com> wrote:
>> > Is this a known issue?
>> > Running 0.6.2 I moved a node to different token and eventually saw
>> > errors in
>> > the log.
>> >
>> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
>> > DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecut=
or
>> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
>> > =A0=A0 =A0 =A0 =A0at
>> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66=
)
>> > =A0=A0 =A0 =A0 =A0at
>> >
>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j=
ava:40)
>> > =A0=A0 =A0 =A0 =A0at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu=
tor.java:886)
>> > =A0=A0 =A0 =A0 =A0at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.=
java:908)
>> > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619)
>> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700
>> > CassandraDaemon.java
>> > (line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116805,5,mai=
n]
>> > java.lang.RuntimeException: Cannot service reads while bootstrapping!
>> > =A0=A0 =A0 =A0 =A0at
>> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66=
)
>> > =A0=A0 =A0 =A0 =A0at
>> >
>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j=
ava:40)
>> > =A0=A0 =A0 =A0 =A0at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu=
tor.java:886)
>> > =A0=A0 =A0 =A0 =A0at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.=
java:908)
>> > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619)
>> > ... many more of those and then...
>> > =A0INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709
>> > StorageService.java (line 181) Bootstrap/move completed! Now serving
>> > reads.
>> >
>> > The move ended up "ok" but during the operation the log was filled wit=
h
>> > those errors and at the end of it the ring state was inconsistent.
>> > If I ask the moved node where it is in the ring it tells me something
>> > but
>> > other nodes tell something else...
>> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h
>> > 192.168.254.58 -p 9004 ring
>> > Address =A0 =A0 =A0 Status =A0 =A0 Load =A0 =A0 =A0 =A0 =A0Range
>> > =A0 =A0 =A0Ring
>> >
>> > 170141183460469231731687303715884105727
>> > 192.168.252.88Up =A0 =A0 =A0 =A0 5.7 GB
>> > =A014131484407726020523932116250949797205 =A0 =A0 |<--|
>> > 192.168.252.124Up =A0 =A0 =A0 =A0 2.44 GB
>> > 56713727820156410577229101238628035242 =A0 =A0 | =A0 ^
>> > 192.168.254.58Up =A0 =A0 =A0 =A0 8.13 GB
>> > 113427455640312821154458202477256070484 =A0 =A0v =A0 |
>> > 192.168.254.57Up =A0 =A0 =A0 =A0 6.52 GB
>> > 113427455640312821154458202477256070485 =A0 =A0| =A0 ^
>> > 192.168.252.125Up =A0 =A0 =A0 =A0 6.52 GB
>> > 141784319550391026443072753096570088105 =A0 =A0v =A0 |
>> > 192.168.254.59Up =A0 =A0 =A0 =A0 1.63 GB
>> > 170141183460469231731687303715884105727 =A0 =A0|-->|
>> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool
>> > -h=A0192.168.252.124=A0-p 9004 ring
>> > Address =A0 =A0 =A0 Status =A0 =A0 Load =A0 =A0 =A0 =A0 =A0Range
>> > =A0 =A0 =A0Ring
>> >
>> > 170141183460469231731687303715884105727
>> > 192.168.252.88Up =A0 =A0 =A0 =A0 5.7 GB
>> > =A014131484407726020523932116250949797205 =A0 =A0 |<--|
>> > 192.168.252.124Up =A0 =A0 =A0 =A0 2.46 GB
>> > 56713727820156410577229101238628035242 =A0 =A0 | =A0 ^
>> > 192.168.254.57Up =A0 =A0 =A0 =A0 6.52 GB
>> > 113427455640312821154458202477256070485 =A0 =A0v =A0 |
>> > 192.168.252.125Up =A0 =A0 =A0 =A0 6.52 GB
>> > 141784319550391026443072753096570088105 =A0 =A0| =A0 ^
>> > 192.168.254.58Up =A0 =A0 =A0 =A0 1.63 GB
>> > 141784319550391026443072753096570088106 =A0 =A0v =A0 |
>> > 192.168.254.59Up =A0 =A0 =A0 =A0 1.63 GB
>> > 170141183460469231731687303715884105727 =A0 =A0|-->|
>> > Restarting the moved node fixes the ring view by other hosts.
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com