Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <CACwRk392ELtWG+3dtPR5Wxc364m9LNyzGXYdae4SVuxQTCEDqg@mail.gmail.com>
References: <CACwRk392ELtWG+3dtPR5Wxc364m9LNyzGXYdae4SVuxQTCEDqg@mail.gmail.com>
From: Alain RODRIGUEZ <arodrime@gmail.com>
Date: Thu, 22 Sep 2016 17:00:59 +0200
Message-ID: <CA+VSrLoET76VO7X-=EON24B57FmVgb=qZgyrQeUZAMgY80F-QQ@mail.gmail.com>
Subject: Re: Rebuild failing when adding new datacenter (3.0.8)
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a113df53c33e872053d19ef90
archived-at: Thu, 22 Sep 2016 15:01:33 -0000

--001a113df53c33e872053d19ef90
Content-Type: text/plain; charset=UTF-8

It could be a bug.

Yet I am not very aware of this system_distributed keyspace, but from what
I see, it is using a simple strategy:

root@tlp-cassandra-2:~# echo "DESCRIBE KEYSPACE system_distributed;" |
cqlsh $(hostname -I | awk '{print $1}')

CREATE KEYSPACE system_distributed WITH replication = {'class':
'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

Let's first check some stuff. Could you share the output of:


   - echo "DESCRIBE KEYSPACE system_distributed;" | cqlsh
   [ip_address_of_the_server]
   - nodetool status
   - nodetool status system_distributed
   - Let us know about the snitch you are using and the corresponding
   configuration.


I am trying to make sure the command you used is expected to work, given
your setup.

My guess is this you might need to alter this keyspace accordingly to your
cluster setup.

Just guessing, hope that helps.

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-09-22 15:47 GMT+02:00 Timo Ahokas <timo.ahokas@gmail.com>:

> Hi,
>
> We have a Cassandra 3.0.8 cluster (recently upgraded from 2.1.15)
> currently running in two data centers (13 and 19 nodes, RF3 in both). We
> are adding a third data center before decommissioning one of the earlier
> ones. Installing Cassandra (3.0.8) goes fine and all the nodes join the
> cluster (not set to bootstrap, as documented in
> https://docs.datastax.com/en/cassandra/3.0/cassandra/
> operations/opsAddDCToCluster.html).
>
> When trying to rebuild nodes in the new DC from a previous DC (nodetool
> rebuild -- DC1), we get the following error:
>
> Unable to find sufficient sources for streaming range (597769692463489739,597931451954862346]
> in keyspace system_distributed
>
> The same error occurs which ever of the 2 existing DCs we try to rebuild
> from.
>
> We run pr repairs (nodetool repair -pr) on all nodes twice a week via cron.
>
> Any advice on how to get the rebuild started?
>
> Best regards,
> Timo
>
>
>
>
>

--001a113df53c33e872053d19ef90
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">It could be a bug.<div><br></div><div>Yet I am not very aw=
are of this system_distributed keyspace, but from what I see, it is using a=
 simple strategy:</div><div><br></div><div>


<p class=3D"gmail-p1"><span class=3D"gmail-s1"><font face=3D"monospace, mon=
ospace" size=3D"1">root@tlp-cassandra-2:~# echo &quot;DESCRIBE KEYSPACE sys=
tem_distributed;&quot; | cqlsh $(hostname -I | awk &#39;{print $1}&#39;)</f=
ont></span></p>
<p class=3D"gmail-p1"><span class=3D"gmail-s1"><font face=3D"monospace, mon=
ospace" size=3D"1">CREATE KEYSPACE system_distributed WITH replication =3D =
{&#39;class&#39;: &#39;SimpleStrategy&#39;, &#39;replication_factor&#39;: &=
#39;3&#39;}=C2=A0 AND durable_writes =3D true;</font></span></p><div><br></=
div><div>Let&#39;s first check some stuff. Could you share the output of:</=
div><div><br></div><div><ul><li>echo &quot;DESCRIBE KEYSPACE system_distrib=
uted;&quot; | cqlsh [ip_address_of_the_server]<br></li><li>nodetool status<=
br></li><li>nodetool status=C2=A0system_distributed<br></li><li>Let us know=
 about the snitch you are using and the corresponding configuration.</li></=
ul><div><br></div></div><div>I am trying to make sure the command you used =
is expected to work, given your setup.</div></div><div><br></div><div>My gu=
ess is this you might need to alter this keyspace accordingly to your clust=
er setup.</div><div><br></div><div>Just guessing, hope that helps.</div><di=
v><br></div><div>C*heers,</div><div><div>-----------------------</div><div>=
Alain Rodriguez - @arodream - <a href=3D"mailto:alain@thelastpickle.com">al=
ain@thelastpickle.com</a></div><div>France</div><div><br></div><div>The Las=
t Pickle - Apache Cassandra Consulting</div><div><a href=3D"http://www.thel=
astpickle.com">http://www.thelastpickle.com</a></div></div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">2016-09-22 15:47 GMT+02:00 =
Timo Ahokas <span dir=3D"ltr">&lt;<a href=3D"mailto:timo.ahokas@gmail.com" =
target=3D"_blank">timo.ahokas@gmail.com</a>&gt;</span>:<br><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad=
ding-left:1ex"><div dir=3D"ltr">Hi,<div><br></div><div>We have a Cassandra =
3.0.8 cluster (recently upgraded from 2.1.15) currently running in two data=
 centers (13 and 19 nodes, RF3 in both). We are adding a third data center =
before decommissioning one of the earlier ones. Installing Cassandra (3.0.8=
) goes fine and all the nodes join the cluster (not set to bootstrap, as do=
cumented in <a href=3D"https://docs.datastax.com/en/cassandra/3.0/cassandra=
/operations/opsAddDCToCluster.html" target=3D"_blank">https://docs.datastax=
.com/en/<wbr>cassandra/3.0/cassandra/<wbr>operations/opsAddDCToCluster.<wbr=
>html</a>).</div><div><br></div><div>When trying to rebuild nodes in the ne=
w DC from a previous DC (nodetool rebuild -- DC1), we get the following err=
or:</div><div><br></div><div>Unable to find sufficient sources for streamin=
g range (597769692463489739,<wbr>597931451954862346] in keyspace system_dis=
tributed<br></div><div><br></div><div>The same error occurs which ever of t=
he 2 existing DCs we try to rebuild from.</div><div><br></div><div>We run p=
r repairs (nodetool repair -pr) on all nodes twice a week via cron.</div><d=
iv><br></div><div>Any advice on how to get the rebuild started?</div><div><=
br></div><div>Best regards,</div><div>Timo</div><div><br></div><div><br></d=
iv><div><br></div><div><br></div></div>
</blockquote></div><br></div>

--001a113df53c33e872053d19ef90--