Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D8E76103AB for ; Sat, 6 Apr 2013 11:55:58 +0000 (UTC) Received: (qmail 64357 invoked by uid 500); 6 Apr 2013 11:55:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 64243 invoked by uid 500); 6 Apr 2013 11:55:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 64228 invoked by uid 99); 6 Apr 2013 11:55:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Apr 2013 11:55:55 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: unknown (athena.apache.org: error in processing during lookup of kais@neteck-fr.com) Received: from [209.85.219.50] (HELO mail-oa0-f50.google.com) (209.85.219.50) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Apr 2013 11:55:52 +0000 Received: by mail-oa0-f50.google.com with SMTP id n1so4804871oag.37 for ; Sat, 06 Apr 2013 04:55:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type:x-gm-message-state; bh=AeDc9rUuF38G1YxKTR3joLEBq+gdD5zH4bS+5pVBR7s=; b=foqU2+CVxe+uhFBcQD673bVG8AXKooMTNWq7UjDu45x0LaLuvI8d3Jtntrq/h9kEVR oSeT2AhUYBZnFv3dqrclfgRMzkLp36qgNRzLSQ6TQfp+yf/QDH+TGwnDshgQYsGq+Jnm UPcZR9Ok5GbAZ7Y3VYYRIPGhDheJGcT5EMrNbTPNQs8onuLDPkuGL7B7sann+5rxw8tq j/4kooXozuPVKCoAqd3u74eonuRdV34AMwRRN7eVJ8UvQEfH6iWSDjVZhWNqXRYqNuiG AQ4RZCEJKTa843C6MkhkbSwGxFGWswsfUqTi6dgCMN7nNr6tkJB/Nf1R2G5b3EfGgcrr jzYA== MIME-Version: 1.0 X-Received: by 10.60.96.10 with SMTP id do10mr4725426oeb.123.1365249330196; Sat, 06 Apr 2013 04:55:30 -0700 (PDT) Received: by 10.182.80.71 with HTTP; Sat, 6 Apr 2013 04:55:29 -0700 (PDT) In-Reply-To: References: <831ABE98-8FC9-4F7A-9E4F-6B24222C90B0@thelastpickle.com> <6B347F2D-B4D0-4954-AEDB-70408089B0C8@thelastpickle.com> Date: Sat, 6 Apr 2013 13:55:29 +0200 Message-ID: Subject: Re: Lost data after expanding cluster c* 1.2.3-1 From: Kais Ahmed To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=e89a8f642c9c5b479504d9afe1e6 X-Gm-Message-State: ALoCoQmw866PUzc3phcLDYw6GXU+oFjA0fL5yEJewRlTl+Sj2TTxc2Hm2ACcJpFxw5Iku/B9qleV X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f642c9c5b479504d9afe1e6 Content-Type: text/plain; charset=ISO-8859-1 hi aaron, nodetool compactionstats on all nodes return 1 pending task : ubuntu@app:~$ nodetool compactionstats host pending tasks: 1 Active compaction remaining time : n/a The command nodetool rebuild_index was launched several days ago. 2013/4/5 aaron morton > but nothing's happening, how can i monitor the progress? and how can i > know when it's finished? > > > check nodetool compacitonstats > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 4/04/2013, at 2:51 PM, Kais Ahmed wrote: > > Hi aaron, > > I ran the command "nodetool rebuild_index host keyspace cf" on all the > nodes, in the log i see : > > INFO [RMI TCP Connection(5422)-10.34.139.xxx] 2013-04-04 08:31:53,641 > ColumnFamilyStore.java (line 558) User Requested secondary index re-build > for ... > > but nothing's happening, how can i monitor the progress? and how can i > know when it's finished? > > Thanks, > > > 2013/4/2 aaron morton > >> The problem come from that i don't put auto_boostrap to true for the new >> nodes, not in this documentation ( >> http://www.datastax.com/docs/1.2/install/expand_ami) >> >> auto_bootstrap defaults to True if not specified in the yaml. >> >> can i do that at any time, or when the cluster are not loaded >> >> Not sure what the question is. >> Both those operations are online operations you can do while the node is >> processing requests. >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 1/04/2013, at 9:26 PM, Kais Ahmed wrote: >> >> > At this moment the errors started, we see that members and other data >> are gone, at this moment the nodetool status return (in red color the 3 new >> nodes) >> > What errors? >> The errors was in my side in the application, not cassandra errors >> >> > I put for each of them seeds = A ip, and start each with two minutes >> intervals. >> > When I'm making changes I tend to change a single node first, confirm >> everything is OK and then do a bulk change. >> Thank you for that advice. >> >> >I'm not sure what or why it went wrong, but that should get you to a >> stable place. If you have any problems keep an eye on the logs for errors >> or warnings. >> The problem come from that i don't put auto_boostrap to true for the new >> nodes, not in this documentation ( >> http://www.datastax.com/docs/1.2/install/expand_ami) >> >> >if you are using secondary indexes use nodetool rebuild_index to rebuild >> those. >> can i do that at any time, or when the cluster are not loaded >> >> Thanks aaron, >> >> 2013/4/1 aaron morton >> >>> Please do not rely on colour in your emails, the best way to get your >>> emails accepted by the Apache mail servers is to use plain text. >>> >>> > At this moment the errors started, we see that members and other data >>> are gone, at this moment the nodetool status return (in red color the 3 new >>> nodes) >>> What errors? >>> >>> > I put for each of them seeds = A ip, and start each with two minutes >>> intervals. >>> When I'm making changes I tend to change a single node first, confirm >>> everything is OK and then do a bulk change. >>> >>> > Now the cluster seem to work normally, but i can use the secondary for >>> the moment, the queryanswer are random >>> run nodetool repair -pr on each node, let it finish before starting the >>> next one. >>> if you are using secondary indexes use nodetool rebuild_index to rebuild >>> those. >>> Add one node new node to the cluster and confirm everything is ok, then >>> add the remaining ones. >>> >>> >I'm not sure what or why it went wrong, but that should get you to a >>> stable place. If you have any problems keep an eye on the logs for errors >>> or warnings. >>> >>> Cheers >>> >>> ----------------- >>> Aaron Morton >>> Freelance Cassandra Consultant >>> New Zealand >>> >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 31/03/2013, at 10:01 PM, Kais Ahmed wrote: >>> >>> > Hi aaron, >>> > >>> > Thanks for reply, i will try to explain what append exactly >>> > >>> > I had 4 C* called [A,B,C,D] cluster (1.2.3-1 version) start with ec2 >>> ami (https://aws.amazon.com/amis/datastax-auto-clustering-ami-2-2) with >>> > this config --clustername myDSCcluster --totalnodes 4--version >>> community >>> > >>> > Two days after this cluster in production, i saw that the cluster was >>> overload, I wanted to extend it by adding 3 another nodes. >>> > >>> > I create a new cluster with 3 C* [D,E,F] ( >>> https://aws.amazon.com/amis/datastax-auto-clustering-ami-2-2) >>> > >>> > And follow the documentation ( >>> http://www.datastax.com/docs/1.2/install/expand_ami) for adding them in >>> the ring. >>> > I put for each of them seeds = A ip, and start each with two minutes >>> intervals. >>> > >>> > At this moment the errors started, we see that members and other data >>> are gone, at this moment the nodetool status return (in red color the 3 new >>> nodes) >>> > >>> > Datacenter: eu-west >>> > =================== >>> > Status=Up/Down >>> > |/ State=Normal/Leaving/Joining/ >>> >> Moving >>> >> -- Address Load Tokens Owns Host ID >>> Rack >>> >> UN 10.34.142.xxx 10.79 GB 256 15.4% >>> 4e2e26b8-aa38-428c-a8f5-e86c13eb4442 1b >>> >> UN 10.32.49.xxx 1.48 MB 256 13.7% >>> e86f67b6-d7cb-4b47-b090-3824a5887145 1b >>> >> UN 10.33.206.xxx 2.19 MB 256 11.9% >>> 92af17c3-954a-4511-bc90-29a9657623e4 1b >>> >> UN 10.32.27.xxx 1.95 MB 256 14.9% >>> 862e6b39-b380-40b4-9d61-d83cb8dacf9e 1b >>> >> UN 10.34.139.xxx 11.67 GB 256 15.5% >>> 0324e394-b65f-46c8-acb4-1e1f87600a2c 1b >>> >> UN 10.34.147.xxx 11.18 GB 256 13.9% >>> cfc09822-5446-4565-a5f0-d25c917e2ce8 1b >>> >> UN 10.33.193.xxx 10.83 GB 256 14.7% >>> 59f440db-cd2d-4041-aab4-fc8e9518c954 1b >>> > >>> > I saw that the 3 nodes have join the ring but they had no data, i put >>> the website in maintenance and lauch a nodetool repair on >>> > the 3 new nodes, during 5 hours i see in opcenter the data streamed to >>> the new nodes (very nice :)) >>> > >>> > During this time, i write a script to check if all members are present >>> (relative to a copy of members in mysql). >>> > >>> > After data streamed seems to be finish, but i'm not sure because >>> nodetool compactionstats show pending task but nodetool netstats seems to >>> be ok. >>> > >>> > I ran my script to check if the data, but members are still missing. >>> > >>> > I decide to roolback by running nodetool decommission node D, E, F >>> > >>> > I re run my script, all seems to be ok but secondary index have >>> strange behavior, >>> > some time the row was returned some times no result. >>> > >>> > the user kais can be retrieve using his key with cassandra-cli but if >>> i use cqlsh : >>> > >>> > cqlsh:database> SELECT login FROM userdata where login='kais' ; >>> > >>> > login >>> > ---------------- >>> > kais >>> > >>> > cqlsh:database> SELECT login FROM userdata where login='kais' ; //empty >>> > cqlsh:database> SELECT login FROM userdata where login='kais' ; >>> > >>> > login >>> > ---------------- >>> > kais >>> > >>> > cqlsh:database> SELECT login FROM userdata where login='kais' ; >>> > >>> > login >>> > ---------------- >>> > kais >>> > >>> > cqlsh:database> SELECT login FROM userdata where login='kais' ; //empty >>> > cqlsh:database> SELECT login FROM userdata where login='kais' ; >>> > >>> > login >>> > ---------------- >>> > kais >>> > >>> > cqlsh:mydatabase>Tracing on; >>> > When tracing is activate i have this error but not all time >>> > cqlsh:mydatabase> SELECT * FROM userdata where login='kais' ; >>> > unsupported operand type(s) for /: 'NoneType' and 'float' >>> > >>> > >>> > NOTE : When the cluster contained 7 nodes, i see that my table >>> userdata (RF 3) on node D was replicated on E and F, that would seem >>> strange because its 3 node was not correctly filled >>> > >>> > Now the cluster seem to work normally, but i can use the secondary for >>> the moment, the query answer are random >>> > >>> > Thanks a lot for any help, >>> > Kais >>> > >>> > >>> > >>> > >>> > >>> > 2013/3/31 aaron morton >>> > First thought is the new nodes were marked as seeds. >>> > Next thought is check the logs for errors. >>> > >>> > You can always run a nodetool repair if you are concerned data is not >>> where you think it should be. >>> > >>> > Cheers >>> > >>> > >>> > ----------------- >>> > Aaron Morton >>> > Freelance Cassandra Consultant >>> > New Zealand >>> > >>> > @aaronmorton >>> > http://www.thelastpickle.com >>> > >>> > On 29/03/2013, at 8:01 PM, Kais Ahmed wrote: >>> > >>> >> Hi all, >>> >> >>> >> I follow this tutorial for expanding a 4 c* cluster (production) and >>> add 3 new nodes. >>> >> >>> >> Datacenter: eu-west >>> >> =================== >>> >> Status=Up/Down >>> >> |/ State=Normal/Leaving/Joining/Moving >>> >> -- Address Load Tokens Owns Host ID >>> Rack >>> >> UN 10.34.142.xxx 10.79 GB 256 15.4% >>> 4e2e26b8-aa38-428c-a8f5-e86c13eb4442 1b >>> >> UN 10.32.49.xxx 1.48 MB 256 13.7% >>> e86f67b6-d7cb-4b47-b090-3824a5887145 1b >>> >> UN 10.33.206.xxx 2.19 MB 256 11.9% >>> 92af17c3-954a-4511-bc90-29a9657623e4 1b >>> >> UN 10.32.27.xxx 1.95 MB 256 14.9% >>> 862e6b39-b380-40b4-9d61-d83cb8dacf9e 1b >>> >> UN 10.34.139.xxx 11.67 GB 256 15.5% >>> 0324e394-b65f-46c8-acb4-1e1f87600a2c 1b >>> >> UN 10.34.147.xxx 11.18 GB 256 13.9% >>> cfc09822-5446-4565-a5f0-d25c917e2ce8 1b >>> >> UN 10.33.193.xxx 10.83 GB 256 14.7% >>> 59f440db-cd2d-4041-aab4-fc8e9518c954 1b >>> >> >>> >> The data are not streamed. >>> >> >>> >> Can any one help me, our web site is down. >>> >> >>> >> Thanks a lot, >>> >> >>> >> >>> > >>> > >>> >>> >> >> > > --e89a8f642c9c5b479504d9afe1e6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable hi aaron,

nodetool compactionstats on all nodes return 1 pending tas= k :

ubuntu@app:~$ nodetool compactionstats host
pending tasks: 1<= br>Active compaction remaining time :=A0=A0=A0=A0=A0=A0=A0 n/a

The c= ommand nodetool rebuild_index was launched several days ago.

2013/4/5 aaron morton <= aaron@thelastp= ickle.com>
but=A0nothing's happening, how can i monitor the progress? a= nd how can i know when it's finished?

<= /div>
check nodetool compacitonstats

Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 4/04/2013, at 2:51 PM, Kais A= hmed <kais@netec= k-fr.com> wrote:

Hi aaron,
I ran the command "nodetool rebuild_index host keyspace cf" on a= ll the nodes, in the log i see :

INFO [RMI TCP Connection(5422)-10.34.139.xxx] 2013-04-04 08:31:53,641 C= olumnFamilyStore.java (line 558) User Requested secondary index re-build fo= r ...

but nothing's happening, how can i monitor the progress? and = how can i know when it's finished?

Thanks,
=A0

=
2013/4/2 aaron morton <aaron@thelastpickl= e.com>
The problem come from that i don't put=A0 auto_b= oostrap to true for the new nodes, not in this documentation (http:/= /www.datastax.com/docs/1.2/install/expand_ami)
auto_bootstrap defaults to True if not specified in the yaml.=A0
=
can i= do=A0that at any time, or when the=A0cluster=A0are not=A0loaded
Not sure what the question is.=A0
Both those operations are online operations you can do while the node is = processing requests.=A0
=A0
Cheers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 1/04/2013, at 9:26 PM, Kais Ahmed <kais@neteck-fr.com&= gt; wrote:

> At this moment the error= s started, we see that=20 members and other data are gone, at this moment the nodetool status=20 return (in red color the 3 new nodes)
> What errors?
The errors wa= s in my side in the application, not cassandra errors

> I put for= each of them seeds =3D A ip, and start each with two minutes intervals. > When I'm making changes I tend to change a single node first, conf= irm everything is OK and then do a bulk change.
Thank you for that advic= e.

>I'm not sure what or why it went wrong, but that should g= et you to a=20 stable place. If you have any problems keep an eye on the logs for=20 errors or warnings.
The problem come from that i don't put=A0 auto_= boostrap to true for the new nodes, not in this documentation (http:= //www.datastax.com/docs/1.2/install/expand_ami)

>if you are using secondary indexes use nodetool rebuild_index to re= build those.
can i do that at any t= ime, or when the cluster are not loaded

Thanks aaron,

2013/4/1 = aaron morton <aaron@thelastpickle.com>
Please do not rely on colour in your emails, the best way to get your email= s accepted by the Apache mail servers is to use plain text.

> At this moment the errors started, we see that members and other data = are gone, at this moment the nodetool status return (in red color the 3 new= nodes)
What errors?

> I put for each of them seeds =3D A ip, and start each with two minutes= intervals.
When I'm making changes I tend to change a single node first, con= firm everything is OK and then do a bulk change.

> Now the cluster seem to work normally, but i can use the secondary for= the moment, the queryanswer are random
run nodetool repair -pr on each node, let it finish before starting t= he next one.
if you are using secondary indexes use nodetool rebuild_index to rebuild th= ose.
Add one node new node to the cluster and confirm everything is ok, then add= the remaining ones.

>I'm not sure what or why it went wrong, but that should get you to = a stable place. If you have any problems keep an eye on the logs for errors= or warnings.

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thel= astpickle.com

On 31/03/2013, at 10:01 PM, Kais Ahmed <kais@neteck-fr.com> wrote:

> Hi aaron,
>
> Thanks for reply, i will try to explain what append exactly
>
> I had 4 C* called [A,B,C,D] cluster (1.2.3-1 version) start with ec2 a= mi (https://aws.amazon.com/amis/datastax-auto-clustering-am= i-2-2) with
> this config --clustername myDSCcluster --totalnodes 4--version communi= ty
>
> Two days after this cluster in production, i saw that the cluster was = overload, I wanted to extend it by adding 3 another nodes.
>
> I create a new cluster with 3 C* [D,E,F] =A0(https://a= ws.amazon.com/amis/datastax-auto-clustering-ami-2-2)
>
> And follow the documentation (http://www.datastax.com/docs/1.2/= install/expand_ami) for adding them in the ring.
> I put for each of them seeds =3D A ip, and start each with two minutes= intervals.
>
> At this moment the errors started, we see that members and other data = are gone, at this moment the nodetool status return (in red color the 3 new= nodes)
>
> Datacenter: eu-west
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Status=3DUp/Down
> |/ State=3DNormal/Leaving/Joining/
>> Moving
>> -- =A0Address =A0 =A0 =A0 =A0 =A0 Load =A0 =A0 =A0 Tokens =A0Owns = =A0 Host ID =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Rac= k
>> UN =A010.34.142.xxx =A0 =A0 10.79 GB =A0 256 =A0 =A0 15.4% =A04e2e= 26b8-aa38-428c-a8f5-e86c13eb4442 =A01b
>> UN =A010.32.49.xxx =A0 =A0 =A0 1.48 MB =A0 =A0256 =A0 =A0 =A0 =A01= 3.7% =A0e86f67b6-d7cb-4b47-b090-3824a5887145 =A01b
>> UN =A010.33.206.xxx =A0 =A0 =A02.19 MB =A0 =A0256 =A0 =A011.9% =A0= 92af17c3-954a-4511-bc90-29a9657623e4 =A01b
>> UN =A010.32.27.xxx =A0 =A0 =A0 1.95 MB =A0 =A0256 =A0 =A0 =A014.9%= =A0862e6b39-b380-40b4-9d61-d83cb8dacf9e =A01b
>> UN =A010.34.139.xxx =A0 =A0 11.67 GB =A0 256 =A0 =A015.5% =A00324e= 394-b65f-46c8-acb4-1e1f87600a2c =A01b
>> UN =A010.34.147.xxx =A0 =A0 11.18 GB =A0 256 =A0 =A0 13.9% =A0cfc0= 9822-5446-4565-a5f0-d25c917e2ce8 =A01b
>> UN =A010.33.193.xxx =A0 =A0 10.83 GB =A0 256 =A0 =A0 =A014.7% =A05= 9f440db-cd2d-4041-aab4-fc8e9518c954 =A01b
>
> I saw that the 3 nodes have join the ring but they had no data, i put = the website in maintenance and lauch a nodetool repair on
> the 3 new nodes, during 5 hours i see in opcenter the data streamed to= the new nodes (very nice :))
>
> During this time, i write a script to check if all members are present= (relative to a copy of members in mysql).
>
> After data streamed seems to be finish, but i'm not sure because n= odetool compactionstats show pending task but nodetool netstats seems to be= ok.
>
> I ran my script to check if the data, but members are still missing. >
> I decide to roolback by running nodetool decommission node D, E, F
>
> I re run my script, all seems to be ok but secondary index have strang= e behavior,
> some time the row was returned some times no result.
>
> the user kais can be retrieve using his key with cassandra-cli but if = i use cqlsh :
>
> cqlsh:database> SELECT login FROM userdata where login=3D'kais&= #39; ;
>
> =A0login
> ----------------
> =A0kais
>
> cqlsh:database> SELECT login FROM userdata where login=3D'kais&= #39; ; //empty
> cqlsh:database> SELECT login FROM userdata where login=3D'kais&= #39; ;
>
> =A0login
> ----------------
> =A0kais
>
> cqlsh:database> SELECT login FROM userdata where login=3D'kais&= #39; ;
>
> =A0login
> ----------------
> =A0kais
>
> cqlsh:database> SELECT login FROM userdata where login=3D'kais&= #39; ; //empty
> cqlsh:database> SELECT login FROM userdata where login=3D'kais&= #39; ;
>
> =A0login
> ----------------
> =A0kais
>
> cqlsh:mydatabase>Tracing on;
> When tracing is activate i have this error but not all time
> cqlsh:mydatabase> SELECT * FROM userdata where login=3D'kais= 9; ;
> unsupported operand type(s) for /: 'NoneType' and 'float&#= 39;
>
>
> NOTE : When the cluster contained 7 nodes, i see that my table userdat= a (RF 3) on node D was replicated on E and F, that would seem strange becau= se its 3 node was not correctly filled
>
> Now the cluster seem to work normally, but i can use the secondary for= the moment, the query answer are random
>
> Thanks a lot for any help,
> Kais
>
>
>
>
>
> 2013/3/31 aaron morton <aaron@thelastpickle.com>
> First thought is the new nodes were marked as seeds.
> Next thought is check the logs for errors.
>
> You can always run a nodetool repair if you are concerned data is not = where you think it should be.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www= .thelastpickle.com
>
> On 29/03/2013, at 8:01 PM, Kais Ahmed <kais@neteck-fr.com> wrote:
>
>> Hi all,
>>
>> I follow this tutorial for expanding a 4 c* cluster (production) a= nd add 3 new nodes.
>>
>> Datacenter: eu-west
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> Status=3DUp/Down
>> |/ State=3DNormal/Leaving/Joining/Moving
>> -- =A0Address =A0 =A0 =A0 =A0 =A0 Load =A0 =A0 =A0 Tokens =A0Owns = =A0 Host ID =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Rac= k
>> UN =A010.34.142.xxx =A0 =A0 10.79 GB =A0 256 =A0 =A0 15.4% =A04e2e= 26b8-aa38-428c-a8f5-e86c13eb4442 =A01b
>> UN =A010.32.49.xxx =A0 =A0 =A0 1.48 MB =A0 =A0256 =A0 =A0 =A0 =A01= 3.7% =A0e86f67b6-d7cb-4b47-b090-3824a5887145 =A01b
>> UN =A010.33.206.xxx =A0 =A0 =A02.19 MB =A0 =A0256 =A0 =A011.9% =A0= 92af17c3-954a-4511-bc90-29a9657623e4 =A01b
>> UN =A010.32.27.xxx =A0 =A0 =A0 1.95 MB =A0 =A0256 =A0 =A0 =A014.9%= =A0862e6b39-b380-40b4-9d61-d83cb8dacf9e =A01b
>> UN =A010.34.139.xxx =A0 =A0 11.67 GB =A0 256 =A0 =A015.5% =A00324e= 394-b65f-46c8-acb4-1e1f87600a2c =A01b
>> UN =A010.34.147.xxx =A0 =A0 11.18 GB =A0 256 =A0 =A0 13.9% =A0cfc0= 9822-5446-4565-a5f0-d25c917e2ce8 =A01b
>> UN =A010.33.193.xxx =A0 =A0 10.83 GB =A0 256 =A0 =A0 =A014.7% =A05= 9f440db-cd2d-4041-aab4-fc8e9518c954 =A01b
>>
>> The data are not streamed.
>>
>> Can any one help me, our web site is down.
>>
>> Thanks a lot,
>>
>>
>
>






--e89a8f642c9c5b479504d9afe1e6--