hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From praveenesh kumar <praveen...@gmail.com>
Subject Re: A question about Balancer in HDFS
Date Fri, 21 Feb 2014 14:55:35 GMT
Cool, thanks for sharing this info. Cheers

Regards
Prav


On Fri, Feb 21, 2014 at 6:53 AM, QShu <shuyunkun@gmail.com> wrote:

> Hi, Prav:
>
> Thanks for your help!
> Now I found out why the balancer stopped working. My Cluster has 2 racks,
> and the balancer follows the rule that can’t change the number of blocks of
> a rack.
> At the same time, the 2 racks is not in the same disk usage. After I
> change all the node into 1 rack, balancer works fine.
>
> Regards
> QiShu
>
>
> 在 2014年2月20日,19:16,praveenesh kumar <praveenesh@gmail.com> 写道:
>
> Have you noticed any balancing in your datanodes by looking at the
> namenode url (where it shows data-node level usage). You should have.
> I think its not stuck, it just waiting for looking for more data blocks
> which it can move around to balance the cluster.
> There is threshold limit (if I remember it correctly), which you can also
> set while running the balancer (otherwise it uses some default value) and
> keeps on running till it reaches that threshold. I think it would be safe
> to just stop the process, as it has done balancing.
>
> Sorry I don't have the exact answer, just sharing my experiences. Someone
> else might throw better light in here.
>
> Regards
> Prav
>
>
> On Thu, Feb 20, 2014 at 7:09 AM, QShu <shuyunkun@gmail.com> wrote:
>
>> Hi, Guys:
>>
>>         The HDFS Balancer will stuck after running a while in my cluster
>> now, the following is the log, after 14:38:52, the Balancer stuccoed and
>> never continue balancing.
>>         Can anybody give me some help, thank you!
>>
>>
>> ==============================================log==============================================
>> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
>> 14/02/20 14:38:25 INFO balancer.Balancer: p         =
>> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
>> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To
>> Move  Bytes Being Moved
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.102:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.23:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.101:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.27:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.105:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.29:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.25:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.26:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.24:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.114:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.107:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.109:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.30:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.115:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.104:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[
>> 172.19.105.102:50010, utilization=42.58400812492816], Source[
>> 172.19.105.23:50010, utilization=41.94449751804609], Source[
>> 172.19.105.101:50010, utilization=49.25646255747681], Source[
>> 172.19.105.27:50010, utilization=42.13637544454659], Source[
>> 172.19.105.29:50010, utilization=42.253685824453484], Source[
>> 172.19.105.25:50010, utilization=41.29288653593026], Source[
>> 172.19.105.26:50010, utilization=42.58689306059978], Source[
>> 172.19.105.24:50010, utilization=42.681174402528306], Source[
>> 172.19.105.30:50010, utilization=42.111934753109566]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized:
>> [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make
>> the cluster balanced.
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.102:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.25:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.26:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.30:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this
>> iteration
>> 2014-2-20 14:38:26                0                  0 B             1.23
>> TB              90 GB
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803
>> from 172.19.105.102:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.114:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224
>> from 172.19.105.102:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591
>> from 172.19.105.101:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.105:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733
>> from 172.19.105.101:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260
>> from 172.19.105.106:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855
>> from 172.19.105.102:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436
>> from 172.19.105.28:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199
>> from 172.19.105.24:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402
>> from 172.19.105.116:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.30:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231
>> from 172.19.105.114:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320
>> from 172.19.105.105:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442
>> from 172.19.105.26:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340
>> from 172.19.105.111:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589
>> from 172.19.105.103:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114
>> from 172.19.105.27:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422
>> from 172.19.105.30:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604
>> from 172.19.105.24:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709
>> from 172.19.105.27:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544
>> from 172.19.105.108:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025
>> from 172.19.105.26:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376
>> from 172.19.105.106:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705
>> from 172.19.105.23:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432
>> from 172.19.105.29:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded.
>> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded.
>> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded.
>> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded.
>
>
>
>

Mime
View raw message