Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 91C9D10759 for ; Tue, 29 Oct 2013 15:48:00 +0000 (UTC) Received: (qmail 55191 invoked by uid 500); 29 Oct 2013 15:40:29 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 53933 invoked by uid 500); 29 Oct 2013 15:39:28 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 53295 invoked by uid 99); 29 Oct 2013 15:39:09 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Oct 2013 15:39:09 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [178.32.98.131] (HELO mo4.mail-out.ovh.net) (178.32.98.131) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Oct 2013 15:39:03 +0000 Received: from mail243.ha.ovh.net (b6.ovh.net [213.186.33.56]) by mo4.mail-out.ovh.net (Postfix) with SMTP id 82F51FFBD4B for ; Tue, 29 Oct 2013 16:39:48 +0100 (CET) Received: from b0.ovh.net (HELO queueout) (213.186.33.50) by b0.ovh.net with SMTP; 29 Oct 2013 17:38:41 +0200 Received: from unknown (HELO ?127.0.0.1?) (cscetbon@atid.fr@83.206.216.177) by ns0.ovh.net with SMTP; 29 Oct 2013 17:38:38 +0200 From: Cyril Scetbon Content-Type: multipart/alternative; boundary="Apple-Mail=_5EFCCDA4-3987-4BAB-A9B5-DE7A6E27DD4A" Message-Id: <18885BE5-1BA5-4C5A-A63C-DC0597007587@free.fr> Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1816\)) Subject: Re: IllegalStateException when bootstrapping new nodes Date: Tue, 29 Oct 2013 16:38:36 +0100 References: <634DD708-655E-428D-A39A-F7CBE09D0F01@free.fr> To: user@cassandra.apache.org In-Reply-To: <634DD708-655E-428D-A39A-F7CBE09D0F01@free.fr> X-Mailer: Apple Mail (2.1816) X-Ovh-Tracer-Id: 15455790972027193302 X-Ovh-Remote: 83.206.216.177 () X-Ovh-Local: 213.186.33.20 (ns0.ovh.net) X-OVH-SPAMSTATE: OK X-OVH-SPAMSCORE: 0 X-OVH-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrfeeiledrgeeiucetufdoteggodetrfcurfhrohhfihhlvgemucfqggfjnecuuegrihhlohhuthemuceftddtnecu X-Spam-Check: DONE|U 0.5/N X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: 0 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrfeeiledrgeeiucetufdoteggodetrfcurfhrohhfihhlvgemucfqggfjnecuuegrihhlohhuthemuceftddtnecu X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_5EFCCDA4-3987-4BAB-A9B5-DE7A6E27DD4A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Sorry but as the link is bad here is the good one : = http://www.sendspace.com/file/7p81lz --=20 Cyril SCETBON On 29 Oct 2013, at 15:38, Cyril Scetbon wrote: > Hi, >=20 > We added yesterday 4 nodes in one of our data-centers but met an = exception on 2 of them : >=20 > java.lang.IllegalStateException: unable to find sufficient sources for = streaming range (6358673204990085385,6363273598121398976] > at = org.apache.cassandra.dht.RangeStreamer.getRangeFetchMap(RangeStreamer.java= :205) > at = org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:129) > at = org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81) > at = org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:= 890) > at = org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.j= ava:659) > at = org.apache.cassandra.service.StorageService.initServer(StorageService.java= :514) > at = org.apache.cassandra.service.StorageService.initServer(StorageService.java= :411) > at = org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:27= 8) > at = org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:311= ) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 57) > at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at = org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)= >=20 > We didn't find the reason why it didn't work but we are wondering if = it's a BUG. We are using Cassandra 1.2.2 and we resolved the issue with = a rolling restart of other nodes in the same DC and maybe with some = luck... We don't know if it's important but we also added the first 2 = new nodes added to the topology file before we successfully bootstrapped = the last 2 nodes.=20 >=20 > FYI, A few weeks ago we added 4 new nodes in another DC with the same = version of Cassandra without any issue. >=20 > Here is the output log with DEBUG level enabled = http://www.sendspace.com/file/7p81lz >=20 > Thanks > --=20 > Cyril SCETBON >=20 --Apple-Mail=_5EFCCDA4-3987-4BAB-A9B5-DE7A6E27DD4A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Sorry = but as the link is bad here is the good one : http://www.sendspace.com/fil= e/7p81lz
Cyril SCETBON

On 29 Oct 2013, at 15:38, Cyril Scetbon <cyril.scetbon@free.fr> = wrote:

Hi,

We added yesterday 4 nodes in = one of our data-centers but met an exception on 2 of them = :

java.lang.IllegalStateException: unable to = find sufficient sources for streaming = range (6358673204990085385,6363273598121398976]
at = org.apache.cassandra.dht.RangeStreamer.getRangeFetchMap(RangeStreamer.java= :205)
= at = org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:129) = at = org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
<= span class=3D"Apple-tab-span" style=3D"white-space:pre"> = at = org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:= 890)
= at = org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.j= ava:659)
= at = org.apache.cassandra.service.StorageService.initServer(StorageService.java= :514)
= at = org.apache.cassandra.service.StorageService.initServer(StorageService.java= :411)
= at = org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:27= 8)
= at = org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:311= )
= at sun.reflect.NativeMethodAccessorImpl.invoke0(Native = Method)
= at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 57)
= at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingM= ethodAccessorImpl.java:43)
at = java.lang.reflect.Method.invoke(Method.java:601)
at = org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)=

We didn't find the reason why it didn't work but we are = wondering if it's a BUG. We are using Cassandra 1.2.2 and we resolved = the issue with a rolling restart of other nodes in the same DC and maybe = with some luck... We don't know if it's important but we also added the = first 2 new nodes added to the topology file before we successfully = bootstrapped the last 2 nodes. 

FYI, A few = weeks ago we added 4 new nodes in another DC with the same version of = Cassandra without any issue.

Here is the output = log with DEBUG level enabled http://www.sendspace.com/file/7p81lz<= /a>



= --Apple-Mail=_5EFCCDA4-3987-4BAB-A9B5-DE7A6E27DD4A--