Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 50497 invoked from network); 5 Jan 2011 13:51:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Jan 2011 13:51:36 -0000 Received: (qmail 18072 invoked by uid 500); 5 Jan 2011 13:51:34 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 17853 invoked by uid 500); 5 Jan 2011 13:51:34 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 17845 invoked by uid 99); 5 Jan 2011 13:51:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 13:51:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rantav@gmail.com designates 209.85.216.172 as permitted sender) Received: from [209.85.216.172] (HELO mail-qy0-f172.google.com) (209.85.216.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 13:51:26 +0000 Received: by qyk34 with SMTP id 34so17433735qyk.10 for ; Wed, 05 Jan 2011 05:51:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:received :in-reply-to:references:date:message-id:subject:from:to:content-type; bh=krN2D5E534Ky9knNSl31tAXPuGGZQ5/LKiDCgKrgXWo=; b=DYyx0NLg9fMWQQ3X2vGPsvK+wjc6dItJ/GN5nV7dVf4Kq/soY2LMMgL2qVv9QPFMny TIY82G5PbxUuCMYOaRskeFuZcVDio4duU36aIfyaalzj0oA405s6dZvLL2/VN1GxCuAD byqJwRh2mdg6rRwoNPtQ7QtQmuxocCqw52+PA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=h/1K0b3OXRIEu9DrGzMGDocEiIYjNr6M+oDGq/HkJuGMRU+0rCq+5AScTyDcFupdH4 XAGNwwuD3yUAOmpgsF5AcKeCf73Z+pg1a5xEurnyMpelKDBzYUMaSXXFParxxL/zglPx jnz/I/8Q7S7yixOYuRO5S6gYM2vsaKiC4GYOw= MIME-Version: 1.0 Received: by 10.229.128.228 with SMTP id l36mr19588248qcs.296.1294235465768; Wed, 05 Jan 2011 05:51:05 -0800 (PST) Received: by 10.229.212.129 with HTTP; Wed, 5 Jan 2011 05:51:05 -0800 (PST) Received: by 10.229.212.129 with HTTP; Wed, 5 Jan 2011 05:51:05 -0800 (PST) In-Reply-To: References: Date: Wed, 5 Jan 2011 15:51:05 +0200 Message-ID: Subject: Re: Bootstrapping taking long From: Ran Tavory To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001517576eb2315f40049919ad00 X-Virus-Checked: Checked by ClamAV on apache.org --001517576eb2315f40049919ad00 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I haven't tried repair. Should I? On Jan 5, 2011 3:48 PM, "Jake Luciani" wrote: > Have you tried not bootstrapping but setting the token and manually calling > repair? > > On Wed, Jan 5, 2011 at 7:07 AM, Ran Tavory wrote: > >> My conclusion is lame: I tried this on several hosts and saw the same >> behavior, the only way I was able to join new nodes was to first start them >> when they are *not in* their own seeds list and after they >> finish transferring the data, then restart them with themselves *in* their >> own seeds list. After doing that the node would join the ring. >> This is either my misunderstanding or a bug, but the only place I found it >> documented stated that the new node should not be in its own seeds list. >> Version 0.6.6. >> >> On Wed, Jan 5, 2011 at 10:35 AM, David Boxenhorn wrote: >> >>> My nodes all have themselves in their list of seeds - always did - and >>> everything works. (You may ask why I did this. I don't know, I must hav= e >>> copied it from an example somewhere.) >>> >>> On Wed, Jan 5, 2011 at 9:42 AM, Ran Tavory wrote: >>> >>>> I was able to make the node join the ring but I'm confused. >>>> What I did is, first when adding the node, this node was not in the seeds >>>> list of itself. AFAIK this is how it's supposed to be. So it was able to >>>> transfer all data to itself from other nodes but then it stayed in the >>>> bootstrapping state. >>>> So what I did (and I don't know why it works), is add this node to the >>>> seeds list in its own storage-conf.xml file. Then restart the server and >>>> then I finally see it in the ring... >>>> If I had added the node to the seeds list of itself when first joining >>>> it, it would not join the ring but if I do it in two phases it did work. >>>> So it's either my misunderstanding or a bug... >>>> >>>> >>>> On Wed, Jan 5, 2011 at 7:14 AM, Ran Tavory wrote: >>>> >>>>> The new node does not see itself as part of the ring, it sees all others >>>>> but itself, so from that perspective the view is consistent. >>>>> The only problem is that the node never finishes to bootstrap. It stays >>>>> in this state for hours (It's been 20 hours now...) >>>>> >>>>> >>>>> $ bin/nodetool -p 9004 -h localhost streams >>>>>> Mode: Bootstrapping >>>>>> Not sending any streams. >>>>>> Not receiving any streams. >>>>> >>>>> >>>>> On Wed, Jan 5, 2011 at 1:20 AM, Nate McCall wrote: >>>>> >>>>>> Does the new node have itself in the list of seeds per chance? This >>>>>> could cause some issues if so. >>>>>> >>>>>> On Tue, Jan 4, 2011 at 4:10 PM, Ran Tavory wrote: >>>>>> > I'm still at lost. I haven't been able to resolve this. I tried >>>>>> > adding another node at a different location on the ring but this node >>>>>> > too remains stuck in the bootstrapping state for many hours withou= t >>>>>> > any of the other nodes being busy with anti compaction or anything >>>>>> > else. I don't know what's keeping it from finishing the bootstrap,no >>>>>> > CPU, no io, files were already streamed so what is it waiting for? >>>>>> > I read the release notes of 0.6.7 and 0.6.8 and there didn't seem to >>>>>> > be anything addressing a similar issue so I figured there was no >>>>>> point >>>>>> > in upgrading. But let me know if you think there is. >>>>>> > Or any other advice... >>>>>> > >>>>>> > On Tuesday, January 4, 2011, Ran Tavory wrote: >>>>>> >> Thanks Jake, but unfortunately the streams directory is empty so = I >>>>>> don't think that any of the nodes is anti-compacting data right now or had >>>>>> been in the past 5 hours. It seems that all the data was already transferred >>>>>> to the joining host but the joining node, after having received the data >>>>>> would still remain in bootstrapping mode and not join the cluster. I'm not >>>>>> sure that *all* data was transferred (perhaps other nodes need to transfer >>>>>> more data) but nothing is actually happening so I assume all has bee= n moved. >>>>>> >> Perhaps it's a configuration error from my part. Should I use I use >>>>>> AutoBootstrap=3Dtrue ? Anything else I should look out for in the >>>>>> configuration file or something else? >>>>>> >> >>>>>> >> >>>>>> >> On Tue, Jan 4, 2011 at 4:08 PM, Jake Luciani >>>>>> wrote: >>>>>> >> >>>>>> >> In 0.6, locate the node doing anti-compaction and look in the >>>>>> "streams" subdirectory in the keyspace data dir to monitor the >>>>>> anti-compaction progress (it puts new SSTables for bootstrapping nod= e in >>>>>> there) >>>>>> >> >>>>>> >> >>>>>> >> On Tue, Jan 4, 2011 at 8:01 AM, Ran Tavory >>>>>> wrote: >>>>>> >> >>>>>> >> >>>>>> >> Running nodetool decommission didn't help. Actually the node refused >>>>>> to decommission itself (b/c it wasn't part of the ring). So I simply stopped >>>>>> the process, deleted all the data directories and started it again. It >>>>>> worked in the sense of the node bootstrapped again but as before, after it >>>>>> had finished moving the data nothing happened for a long time (I'm still >>>>>> waiting, but nothing seems to be happening). >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> Any hints how to analyze a "stuck" bootstrapping node??thanks >>>>>> >> On Tue, Jan 4, 2011 at 1:51 PM, Ran Tavory >>>>>> wrote: >>>>>> >> Thanks Shimi, so indeed anticompaction was run on one of the othe= r >>>>>> nodes from the same DC but to my understanding it has already ended. A few >>>>>> hour ago... >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> I plenty of log messages such as [1] which ended a couple of hour= s >>>>>> ago, and I've seen the new node streaming and accepting the data fro= m the >>>>>> node which performed the anticompaction and so far it was normal so it >>>>>> seemed that data is at its right place. But now the new node seems sort of >>>>>> stuck. None of the other nodes is anticompacting right now or had been >>>>>> anticompacting since then. >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> The new node's CPU is close to zero, it's iostats are almost zero so >>>>>> I can't find another bottleneck that would keep it hanging. >>>>>> >> On the IRC someone suggested I'd maybe retry to join this node, >>>>>> e.g. decommission and rejoin it again. I'll try it now... >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> [1] INFO [COMPACTION-POOL:1] 2011-01-04 04:04:09,721 >>>>>> CompactionManager.java (line 338) AntiCompacting >>>>>> [org.apache.cassandra.io.SSTableReader(path=3D'/outbrain/cassandra/data/out= brain_kvdb/KvAds-6449-Data.db')] >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:18,683 >>>>>> CompactionManager.java (line 338) AntiCompacting >>>>>> [org.apache.cassandra.io.SSTableReader(path=3D'/outbrain/cassandra/data/out= brain_kvdb/KvImpressions-3874-Data.db'),org.apache.cassandra.io.SSTableRead= er(path=3D'/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3873-Data.d= b'),org.apache.cassandra.io.SSTableReader(path=3D'/outbrain/cassandra/data/= outbrain_kvdb/KvImpressions-3876-Data.db')] >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:19,132 >>>>>> CompactionManager.java (line 338) AntiCompacting >>>>>> [org.apache.cassandra.io.SSTableReader(path=3D'/outbrain/cassandra/data/out= brain_kvdb/KvRatings-951-Data.db'),org.apache.cassandra.io.SSTableReader(pa= th=3D'/outbrain/cassandra/data/outbrain_kvdb/KvRatings-976-Data.db'),org.ap= ache.cassandra.io.SSTableReader(path=3D'/outbrain/cassandra/data/outbrain_k= vdb/KvRatings-978-Data.db')] >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:26,486 >>>>>> CompactionManager.java (line 338) AntiCompacting >>>>>> [org.apache.cassandra.io.SSTableReader(path=3D'/outbrain/cassandra/data/out= brain_kvdb/KvAds-6449-Data.db')] >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> On Tue, Jan 4, 2011 at 12:45 PM, shimi wrote: >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> In my experience most of the time it takes for a node to join the >>>>>> cluster is the anticompaction on the other nodes. The streaming part is very >>>>>> fast. >>>>>> >> Check the other nodes logs to see if there is any node doing >>>>>> anticompaction.I don't remember how much data I had in the cluster when I >>>>>> needed to add/remove nodes. I do remember that it took a few hours. >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> The node will join the ring only when it will finish the bootstrap. >>>>>> >> -- >>>>>> >> /Ran >>>>>> >> >>>>>> >> >>>>>> > >>>>>> > -- >>>>>> > /Ran >>>>>> > >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> /Ran >>>>> >>>>> >>>> >>>> >>>> -- >>>> /Ran >>>> >>>> >>> >> >> >> -- >> /Ran >> >> --001517576eb2315f40049919ad00 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

I haven't tried repair.=C2=A0 Should I?

On Jan 5, 2011 3:48 PM, "Jake Luciani"= <jakers@gmail.com> wrote:> Have you tried not bootstrapping but setting the= token and manually calling
> repair?
>
> On Wed, Jan 5, 2011 at 7:07 AM, Ran Tavory &l= t;rantav@gmail.com> wrote:
&g= t;
>> My conclusion is lame: I tried this on several hosts and sa= w the same
>> behavior, the only way I was able to join new nodes was to first s= tart them
>> when they are *not in* their own seeds list and after= they
>> finish transferring the data, then restart them with them= selves *in* their
>> own seeds list. After doing that the node would join the ring.
= >> This is either my misunderstanding or a bug, but the only place I = found it
>> documented stated that the new node should not be in i= ts own seeds list.
>> Version 0.6.6.
>>
>> On Wed, Jan 5, 2011 at 10:3= 5 AM, David Boxenhorn <david@lookin= 2.com>wrote:
>>
>>> My nodes all have themselve= s in their list of seeds - always did - and
>>> everything works. (You may ask why I did this. I don't kno= w, I must have
>>> copied it from an example somewhere.)
>= ;>>
>>> On Wed, Jan 5, 2011 at 9:42 AM, Ran Tavory <rantav@gmail.com> wrote:
>>>
>>>> I was able to make the node join the ring = but I'm confused.
>>>> What I did is, first when adding = the node, this node was not in the seeds
>>>> list of itself= . AFAIK this is how it's supposed to be. So it was able to
>>>> transfer all data to itself from other nodes but then it s= tayed in the
>>>> bootstrapping state.
>>>> S= o what I did (and I don't know why it works), is add this node to the >>>> seeds list in its own storage-conf.xml file. Then restart = the server and
>>>> then I finally see it in the ring...
= >>>> If I had added the node to the seeds list of itself when f= irst joining
>>>> it, it would not join the ring but if I do it in two phase= s it did work.
>>>> So it's either my misunderstanding o= r a bug...
>>>>
>>>>
>>>> On W= ed, Jan 5, 2011 at 7:14 AM, Ran Tavory <rantav@gmail.com> wrote:
>>>>
>>>>> The new node does not see itself a= s part of the ring, it sees all others
>>>>> but itself, = so from that perspective the view is consistent.
>>>>> Th= e only problem is that the node never finishes to bootstrap. It stays
>>>>> in this state for hours (It's been 20 hours now...= )
>>>>>
>>>>>
>>>>> $= bin/nodetool -p 9004 -h localhost streams
>>>>>> Mode= : Bootstrapping
>>>>>> Not sending any streams.
>>>>>&g= t; Not receiving any streams.
>>>>>
>>>>&g= t;
>>>>> On Wed, Jan 5, 2011 at 1:20 AM, Nate McCall <= nate@riptano.com> wrote:
>>>>>
>>>>>> Does the new node have its= elf in the list of seeds per chance? This
>>>>>> could= cause some issues if so.
>>>>>>
>>>>&g= t;> On Tue, Jan 4, 2011 at 4:10 PM, Ran Tavory <rantav@gmail.com> wrote:
>>>>>> > I'm still at lost. I haven't been a= ble to resolve this. I tried
>>>>>> > adding anothe= r node at a different location on the ring but this node
>>>>= ;>> > too remains stuck in the bootstrapping state for many hours = without
>>>>>> > any of the other nodes being busy with anti c= ompaction or anything
>>>>>> > else. I don't kn= ow what's keeping it from finishing the bootstrap,no
>>>>= ;>> > CPU, no io, files were already streamed so what is it waitin= g for?
>>>>>> > I read the release notes of 0.6.7 and 0.6.8 a= nd there didn't seem to
>>>>>> > be anything ad= dressing a similar issue so I figured there was no
>>>>>&= gt; point
>>>>>> > in upgrading. But let me know if you think th= ere is.
>>>>>> > Or any other advice...
>>= >>>> >
>>>>>> > On Tuesday, January = 4, 2011, Ran Tavory <rantav@gmail.co= m> wrote:
>>>>>> >> Thanks Jake, but unfortunately the stream= s directory is empty so I
>>>>>> don't think that = any of the nodes is anti-compacting data right now or had
>>>&g= t;>> been in the past 5 hours. It seems that all the data was already= transferred
>>>>>> to the joining host but the joining node, after ha= ving received the data
>>>>>> would still remain in bo= otstrapping mode and not join the cluster. I'm not
>>>>&= gt;> sure that *all* data was transferred (perhaps other nodes need to t= ransfer
>>>>>> more data) but nothing is actually happening so I = assume all has been moved.
>>>>>> >> Perhaps it&= #39;s a configuration error from my part. Should I use I use
>>>= ;>>> AutoBootstrap=3Dtrue ? Anything else I should look out for in= the
>>>>>> configuration file or something else?
>>&= gt;>>> >>
>>>>>> >>
>>&g= t;>>> >> On Tue, Jan 4, 2011 at 4:08 PM, Jake Luciani <jakers@gmail.com>
>>>>>> wrote:
>>>>>> >>
>= ;>>>>> >> In 0.6, locate the node doing anti-compactio= n and look in the
>>>>>> "streams" subdirect= ory in the keyspace data dir to monitor the
>>>>>> anti-compaction progress (it puts new SSTables for= bootstrapping node in
>>>>>> there)
>>>&g= t;>> >>
>>>>>> >>
>>>>= ;>> >> On Tue, Jan 4, 2011 at 8:01 AM, Ran Tavory <rantav@gmail.com>
>>>>>> wrote:
>>>>>> >>
>= ;>>>>> >>
>>>>>> >> Running= nodetool decommission didn't help. Actually the node refused
>&g= t;>>>> to decommission itself (b/c it wasn't part of the ri= ng). So I simply stopped
>>>>>> the process, deleted all the data directories and = started it again. It
>>>>>> worked in the sense of the= node bootstrapped again but as before, after it
>>>>>>= ; had finished moving the data nothing happened for a long time (I'm st= ill
>>>>>> waiting, but nothing seems to be happening).
&g= t;>>>>> >>
>>>>>> >>
>= ;>>>>> >>
>>>>>> >>
>>>>>> >> Any hints how to analyze a "stuck&qu= ot; bootstrapping node??thanks
>>>>>> >> On Tue,= Jan 4, 2011 at 1:51 PM, Ran Tavory <rantav@gmail.com>
>>>>>> wrote:
>>>>>> >> Thanks= Shimi, so indeed anticompaction was run on one of the other
>>>= ;>>> nodes from the same DC but to my understanding it has already= ended. A few
>>>>>> hour ago...
>>>>>> >>>>>>>> >>
>>>>>> >>>>>>>> >> I plenty of log messages such as [1] whi= ch ended a couple of hours
>>>>>> ago, and I've seen the new node streaming and = accepting the data from the
>>>>>> node which performe= d the anticompaction and so far it was normal so it
>>>>>= > seemed that data is at its right place. But now the new node seems sor= t of
>>>>>> stuck. None of the other nodes is anticompacting r= ight now or had been
>>>>>> anticompacting since then.=
>>>>>> >>
>>>>>> >><= br> >>>>>> >>
>>>>>> >>
&= gt;>>>>> >> The new node's CPU is close to zero, i= t's iostats are almost zero so
>>>>>> I can't = find another bottleneck that would keep it hanging.
>>>>>> >> On the IRC someone suggested I'd mayb= e retry to join this node,
>>>>>> e.g. decommission an= d rejoin it again. I'll try it now...
>>>>>> >&= gt;
>>>>>> >>
>>>>>> >>
&= gt;>>>>> >>
>>>>>> >>
&g= t;>>>>> >>
>>>>>> >> [1] IN= FO [COMPACTION-POOL:1] 2011-01-04 04:04:09,721
>>>>>> CompactionManager.java (line 338) AntiCompacting>>>>>> [org.apache.cassandra.io.SSTableReader(path=3D&#= 39;/outbrain/cassandra/data/outbrain_kvdb/KvAds-6449-Data.db')]
>= >>>>> >>
>>>>>> >>
>>>>>> >>
&= gt;>>>>> >>
>>>>>> >> INFO= [COMPACTION-POOL:1] 2011-01-04 04:34:18,683
>>>>>> Co= mpactionManager.java (line 338) AntiCompacting
>>>>>> [org.apache.cassandra.io.SSTableReader(path=3D'= ;/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3874-Data.db'),or= g.apache.cassandra.io.SSTableReader(path=3D'/outbrain/cassandra/data/ou= tbrain_kvdb/KvImpressions-3873-Data.db'),org.apache.cassandra.io.SSTabl= eReader(path=3D'/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-38= 76-Data.db')]
>>>>>> >>
>>>>>> >>
&= gt;>>>>> >>
>>>>>> >>
&g= t;>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:= 19,132
>>>>>> CompactionManager.java (line 338) AntiCompacting>>>>>> [org.apache.cassandra.io.SSTableReader(path=3D&#= 39;/outbrain/cassandra/data/outbrain_kvdb/KvRatings-951-Data.db'),org.a= pache.cassandra.io.SSTableReader(path=3D'/outbrain/cassandra/data/outbr= ain_kvdb/KvRatings-976-Data.db'),org.apache.cassandra.io.SSTableReader(= path=3D'/outbrain/cassandra/data/outbrain_kvdb/KvRatings-978-Data.db= 9;)]
>>>>>> >>
>>>>>> >>
&= gt;>>>>> >>
>>>>>> >>
&g= t;>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:= 26,486
>>>>>> CompactionManager.java (line 338) AntiCompacting>>>>>> [org.apache.cassandra.io.SSTableReader(path=3D&#= 39;/outbrain/cassandra/data/outbrain_kvdb/KvAds-6449-Data.db')]
>= >>>>> >>
>>>>>> >>
>>>>>> >>
&= gt;>>>>> >>
>>>>>> >>
&g= t;>>>>> >> On Tue, Jan 4, 2011 at 12:45 PM, shimi <= shimi.k@gmail.com> wrote:
>>>>>> >>
>>>>>> >>
&= gt;>>>>> >>
>>>>>> >>
&g= t;>>>>> >>
>>>>>> >> In my = experience most of the time it takes for a node to join the
>>>>>> cluster is the anticompaction on the other nodes. = The streaming part is very
>>>>>> fast.
>>>= ;>>> >> Check the other nodes logs to see if there is any no= de doing
>>>>>> anticompaction.I don't remember how much data = I had in the cluster when I
>>>>>> needed to add/remov= e nodes. I do remember that it took a few hours.
>>>>>>= ; >>
>>>>>> >>
>>>>>> >>
&= gt;>>>>> >>
>>>>>> >>
&g= t;>>>>> >>
>>>>>> >> The no= de will join the ring only when it will finish the bootstrap.
>>>>>> >> --
>>>>>> >> /= Ran
>>>>>> >>
>>>>>> >&g= t;
>>>>>> >
>>>>>> > --
>>>>>> > /Ran
>>>>>> >
>= >>>>>
>>>>>
>>>>>
>= ;>>>>
>>>>> --
>>>>> /Ran >>>>>
>>>>>
>>>>
>>= ;>>
>>>> --
>>>> /Ran
>>>&g= t;
>>>>
>>>
>>
>>
>> = --
>> /Ran
>>
>>
--001517576eb2315f40049919ad00--