From user-return-8228-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Aug 04 19:51:06 2010 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 81179 invoked from network); 4 Aug 2010 19:51:06 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 4 Aug 2010 19:51:06 -0000 Received: (qmail 63763 invoked by uid 500); 4 Aug 2010 19:51:04 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 63729 invoked by uid 500); 4 Aug 2010 19:51:04 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 63721 invoked by uid 99); 4 Aug 2010 19:51:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 19:51:04 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rantav@gmail.com designates 209.85.161.44 as permitted sender) Received: from [209.85.161.44] (HELO mail-fx0-f44.google.com) (209.85.161.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 19:50:59 +0000 Received: by fxm16 with SMTP id 16so1856988fxm.31 for ; Wed, 04 Aug 2010 12:50:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type; bh=+haBTY2KSMV13cYdw8JEtonKO8BNrsWID2Wd0jDsbB8=; b=jBvp+km1qtZUTsP3xcNgtj6EDmnVw28JwceFbUTr23fkAz/k85ymcIf+TKtNRiva3P YHGAaR/S80UwSrwLoe44GA0fIa4ub+dku50C1y9UtsHJG83EUSp42VYZiGV7hnqknVAo CyZm7GEyuTI+5psWqMZ0ZFE5wagMTj1axJKK0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=vg34bUfY6FQ5TrpGtIjSYcqq3N+3IcwVtrrXvzErfp2d+UG49cLjvEqoIHLalUtH4/ m6/lhwdVvSQYWRH7OqGTRK7uw8apZ8aCy7F3Fibv7+vp2yxcvDYsFkgSEtJh7R5sXU5P NcLp7Z30y/auBI3cZWP4xu5q2hsYGliAipKJA= Received: by 10.239.147.212 with SMTP id c20mr418988hbb.173.1280951437204; Wed, 04 Aug 2010 12:50:37 -0700 (PDT) MIME-Version: 1.0 Received: by 10.239.159.4 with HTTP; Wed, 4 Aug 2010 12:50:17 -0700 (PDT) In-Reply-To: References: From: Ran Tavory Date: Wed, 4 Aug 2010 22:50:17 +0300 Message-ID: Subject: Re: RuntimeException: Cannot service reads while bootstrapping! To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016363b9e0063a355048d04bf96 --0016363b9e0063a355048d04bf96 Content-Type: text/plain; charset=UTF-8 reported https://issues.apache.org/jira/browse/CASSANDRA-1359 On Wed, Aug 4, 2010 at 9:51 PM, Jonathan Ellis wrote: > That's definitely a bug. > > On Wed, Aug 4, 2010 at 2:13 PM, Ran Tavory wrote: > > ok, so I don't send writes to bootstrapping or decommissioned nodes, > that's > > cool, but what about the inconsistent ring view after nodetool move, > isn't > > this strange? > > After the move, the moved node has the correct view of the ring but all > > other nodes have the old view. I waited a few minutes after the log said > > that Bootstrap/move completed! Now serving reads but this didn't help, > view > > was still inconsistent. Only restarting the moved node helped other nodes > > realize the change. > > On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis > wrote: > >> > >> Don't point clients at nodes that aren't part of the ring. Cassandra > >> rejecting requests when you do is a "feature." > >> > >> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory wrote: > >> > Is this a known issue? > >> > Running 0.6.2 I moved a node to different token and eventually saw > >> > errors in > >> > the log. > >> > > >> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699 > >> > DebuggableThreadPoolExecutor.java (line 101) Error in > ThreadPoolExecutor > >> > java.lang.RuntimeException: Cannot service reads while bootstrapping! > >> > at > >> > > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66) > >> > at > >> > > >> > > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) > >> > at > >> > > >> > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> > at > >> > > >> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> > at java.lang.Thread.run(Thread.java:619) > >> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700 > >> > CassandraDaemon.java > >> > (line 82) Fatal exception in thread > Thread[ROW-READ-STAGE:116805,5,main] > >> > java.lang.RuntimeException: Cannot service reads while bootstrapping! > >> > at > >> > > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:66) > >> > at > >> > > >> > > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) > >> > at > >> > > >> > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> > at > >> > > >> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> > at java.lang.Thread.run(Thread.java:619) > >> > ... many more of those and then... > >> > INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,709 > >> > StorageService.java (line 181) Bootstrap/move completed! Now serving > >> > reads. > >> > > >> > The move ended up "ok" but during the operation the log was filled > with > >> > those errors and at the end of it the ring state was inconsistent. > >> > If I ask the moved node where it is in the ring it tells me something > >> > but > >> > other nodes tell something else... > >> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -h > >> > 192.168.254.58 -p 9004 ring > >> > Address Status Load Range > >> > Ring > >> > > >> > 170141183460469231731687303715884105727 > >> > 192.168.252.88Up 5.7 GB > >> > 14131484407726020523932116250949797205 |<--| > >> > 192.168.252.124Up 2.44 GB > >> > 56713727820156410577229101238628035242 | ^ > >> > 192.168.254.58Up 8.13 GB > >> > 113427455640312821154458202477256070484 v | > >> > 192.168.254.57Up 6.52 GB > >> > 113427455640312821154458202477256070485 | ^ > >> > 192.168.252.125Up 6.52 GB > >> > 141784319550391026443072753096570088105 v | > >> > 192.168.254.59Up 1.63 GB > >> > 170141183460469231731687303715884105727 |-->| > >> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool > >> > -h 192.168.252.124 -p 9004 ring > >> > Address Status Load Range > >> > Ring > >> > > >> > 170141183460469231731687303715884105727 > >> > 192.168.252.88Up 5.7 GB > >> > 14131484407726020523932116250949797205 |<--| > >> > 192.168.252.124Up 2.46 GB > >> > 56713727820156410577229101238628035242 | ^ > >> > 192.168.254.57Up 6.52 GB > >> > 113427455640312821154458202477256070485 v | > >> > 192.168.252.125Up 6.52 GB > >> > 141784319550391026443072753096570088105 | ^ > >> > 192.168.254.58Up 1.63 GB > >> > 141784319550391026443072753096570088106 v | > >> > 192.168.254.59Up 1.63 GB > >> > 170141183460469231731687303715884105727 |-->| > >> > Restarting the moved node fixes the ring view by other hosts. > >> > > >> > > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of Riptano, the source for professional Cassandra support > >> http://riptano.com > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com > --0016363b9e0063a355048d04bf96 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
reported=C2=A0https://issues.apache.org/jira/browse/CASSANDRA-1359

On Wed, Aug 4, 2010 at 9:51 PM, Jonath= an Ellis <jbellis= @gmail.com> wrote:
That's definitely a bug.

On Wed, Aug 4, 2010 at 2:13 PM, Ran Tavory <rantav@gmail.com> wrote:
> ok, so I don't send writes to bootstrapping or=C2=A0decommissioned= =C2=A0nodes, that's
> cool, but what about the inconsistent ring view after nodetool move, i= sn't
> this strange?
> After the move, the moved node has the correct view of the ring but al= l
> other nodes have the old view. I waited a few minutes after the log sa= id
> that=C2=A0Bootstrap/move completed! Now serving reads but this didn= 9;t help, view
> was still inconsistent. Only=C2=A0restarting=C2=A0the moved node helpe= d other nodes
> realize the change.
> On Wed, Aug 4, 2010 at 3:24 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> Don't point clients at nodes that aren't part of the ring.= =C2=A0Cassandra
>> rejecting requests when you do is a "feature."
>>
>> On Wed, Aug 4, 2010 at 6:52 AM, Ran Tavory <rantav@gmail.com> wrote:
>> > Is this a known issue?
>> > Running 0.6.2 I moved a node to different token and eventuall= y saw
>> > errors in
>> > the log.
>> >
>> > ERROR [ROW-READ-STAGE:116804] 2010-08-04 06:34:29,699
>> > DebuggableThreadPoolExecutor.java (line 101) Error in ThreadP= oolExecutor
>> > java.lang.RuntimeException: Cannot service reads while bootst= rapping!
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandle= r.java:66)
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> >
>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliv= eryTask.java:40)
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Thread= PoolExecutor.java:886)
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPool= Executor.java:908)
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thre= ad.java:619)
>> > ERROR [ROW-READ-STAGE:116805] 2010-08-04 06:34:29,700
>> > CassandraDaemon.java
>> > (line 82) Fatal exception in thread Thread[ROW-READ-STAGE:116= 805,5,main]
>> > java.lang.RuntimeException: Cannot service reads while bootst= rapping!
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandle= r.java:66)
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> >
>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliv= eryTask.java:40)
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Thread= PoolExecutor.java:886)
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPool= Executor.java:908)
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thre= ad.java:619)
>> > ... many more of those and then...
>> > =C2=A0INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-08-04 06:34:29,= 709
>> > StorageService.java (line 181) Bootstrap/move completed! Now = serving
>> > reads.
>> >
>> > The move ended up "ok" but during the operation the= log was filled with
>> > those errors and at the end of it the ring state was inconsis= tent.
>> > If I ask the moved node where it is in the ring it tells me s= omething
>> > but
>> > other nodes tell something else...
>> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool -= h
>> > 192.168.254.58 -p 9004 ring
>> > Address =C2=A0 =C2=A0 =C2=A0 Status =C2=A0 =C2=A0 Load =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0Range
>> > =C2=A0 =C2=A0 =C2=A0Ring
>> >
>> > 170141183460469231731687303715884105727
>> > 192.168.252.88Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 5.7 GB
>> > =C2=A014131484407726020523932116250949797205 =C2=A0 =C2=A0 |&= lt;--|
>> > 192.168.252.124Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 2.44 GB
>> > 56713727820156410577229101238628035242 =C2=A0 =C2=A0 | =C2=A0= ^
>> > 192.168.254.58Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 8.13 GB
>> > 113427455640312821154458202477256070484 =C2=A0 =C2=A0v =C2=A0= |
>> > 192.168.254.57Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 6.52 GB
>> > 113427455640312821154458202477256070485 =C2=A0 =C2=A0| =C2=A0= ^
>> > 192.168.252.125Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 6.52 GB
>> > 141784319550391026443072753096570088105 =C2=A0 =C2=A0v =C2=A0= |
>> > 192.168.254.59Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.63 GB
>> > 170141183460469231731687303715884105727 =C2=A0 =C2=A0|-->|=
>> > (ob1124)(cassandra@cass24:apache-cassandra-0.6.2)$ nodetool >> > -h=C2=A0192.168.252.124=C2=A0-p 9004 ring
>> > Address =C2=A0 =C2=A0 =C2=A0 Status =C2=A0 =C2=A0 Load =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0Range
>> > =C2=A0 =C2=A0 =C2=A0Ring
>> >
>> > 170141183460469231731687303715884105727
>> > 192.168.252.88Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 5.7 GB
>> > =C2=A014131484407726020523932116250949797205 =C2=A0 =C2=A0 |&= lt;--|
>> > 192.168.252.124Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 2.46 GB
>> > 56713727820156410577229101238628035242 =C2=A0 =C2=A0 | =C2=A0= ^
>> > 192.168.254.57Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 6.52 GB
>> > 113427455640312821154458202477256070485 =C2=A0 =C2=A0v =C2=A0= |
>> > 192.168.252.125Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 6.52 GB
>> > 141784319550391026443072753096570088105 =C2=A0 =C2=A0| =C2=A0= ^
>> > 192.168.254.58Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.63 GB
>> > 141784319550391026443072753096570088106 =C2=A0 =C2=A0v =C2=A0= |
>> > 192.168.254.59Up =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.63 GB
>> > 170141183460469231731687303715884105727 =C2=A0 =C2=A0|-->|=
>> > Restarting the moved node fixes the ring view by other hosts.=
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra suppo= rt
>> http://riptano.co= m
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

--0016363b9e0063a355048d04bf96--