hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Lilley <john.lil...@redpoint.net>
Subject RE: How can I increase the speed balancing?
Date Wed, 03 Sep 2014 12:08:23 GMT
I have also found that neither
dfsadmin - setBalanacerBandwidth
nor
dfs.datanode.balance.bandwidthPerSec’
have any notable effect on apparent balancer rate.  This is on Hadoop 2.2.0

john

From: cho ju il [mailto:tjstory@kgrid.co.kr]
Sent: Wednesday, September 03, 2014 12:55 AM
To: user@hadoop.apache.org
Subject: RE: How can I increase the speed balancing?


Bandwidth is enough

And i use command "bin/hdfs dfsadmin -setBalancerBandwidth 52428800 ".

Yet balancing slow.

I think because the file transfer speed is slow and  move 5 files per 1 server .

-----Original Message-----
From: "Srikanth upputuri"<srikanth.upputuri@huawei.com>
To: "user@hadoop.apache.org"<user@hadoop.apache.org>;
Cc:
Sent: 2014-09-03 (수) 14:10:24
Subject: RE: How can I increase the speed balancing?

I am not sure what you meant by ‘Bandwidth is not a lack of data nodes’ but have you configured
the balancer bandwidth property ‘dfs.datanode.balance.bandwidthPerSec’? If not it defaults
to 1KB/s. You can increase this to improve the balancer speed. You may also set it dynamically
using the command ‘dfsadmin -setBalanacerBandwidth newbandwidth’ before running balancer.





From: cho ju il [mailto:tjstory@kgrid.co.kr]
Sent: 03 September 2014 06:31
To: user@hadoop.apache.org
Subject: How can I increase the speed balancing?



hadoop version 2.4.1

Balancing speed is slow.

I think because the file transfer speed is slow.

Bandwidth is not a lack of data nodes.

How can I increase the speed balancing?









*** balancer log

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.207:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.205:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.203:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.210:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.211:40010

2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.114:40010

2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.206:40010

2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.201:40010

2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.202:40010

2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.204:40010

2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 over-utilized:
[Source[192.168.0.203:40010, utilization=99.99693907952093], Source[192.168.0.201:40010, utilization=99.99713240471648],
Source[192.168.0.202:40010, utilization=99.99652052169367]]

2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 underutilized:
[BalancerDatanode[192.168.0.211:40010, utilization=62.735024524531006], BalancerDatanode[192.168.0.114:40010,
utilization=2.3174560700459224]]

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move
70.30 TB to make the cluster balanced.

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move
10 GB bytes from 192.168.0.203:40010 to 192.168.0.211:40010

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move
10 GB bytes from 192.168.0.201:40010 to 192.168.0.114:40010

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 20
GB in this iteration

2014-09-03 09:44:23,643 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216256_3475617 with size=1746577 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.205:40010

2014-09-03 09:45:38,730 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1076170742_2430103 with size=16746627 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.203:40010

2014-09-03 09:46:42,649 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216315_3475676 with size=38338949 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.206:40010

2014-09-03 09:50:13,239 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1074229029_488207 with size=89080307 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.203:40010

2014-09-03 09:50:58,659 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216317_3475678 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.210:40010

2014-09-03 09:51:15,660 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1075275002_1534363 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.204:40010

2014-09-03 09:51:19,661 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216257_3475618 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.202:40010

2014-09-03 09:51:54,243 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1076170740_2430101 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.203:40010

2014-09-03 09:52:15,242 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1074229048_488226 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.205:40010

2014-09-03 09:52:22,744 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1076170741_2430102 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.203:40010



[Image removed by sender.]



[Image removed by sender.]


Mime
View raw message