hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Lilley <john.lil...@redpoint.net>
Subject RE: How can I increase the speed balancing?
Date Tue, 16 Sep 2014 11:17:50 GMT
Srikanth,
The cluster is idle while balancing, and it seems to move about 2MB/minute.  There is no discernable
CPU load.
john

From: Srikanth upputuri [mailto:srikanth.upputuri@huawei.com]
Sent: Thursday, September 04, 2014 12:57 AM
To: user@hadoop.apache.org
Subject: RE: How can I increase the speed balancing?

AFAIK, this setting is meant to throttle bandwidth usage by balancer so that the balancing
traffic will not severely impact the performance of the running jobs. Increasing this value
will show effect only when there is enough total available bandwidth on the network. On an
already overloaded network changing this value may not show much improvement. I suggest you
look at the total network capacity, network usage on your datanodes to assess if there is
sufficient room to increase your balancer bandwidth. Can you also try running balancer when
there is no other traffic and see if changing this value has any impact.

Correction: Default balancer bandwidth is 1MB/s, not 1KB/s as I mentioned in my previous post.
Sorry for the typo.

From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: 03 September 2014 17:38
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: How can I increase the speed balancing?

I have also found that neither
dfsadmin - setBalanacerBandwidth
nor
dfs.datanode.balance.bandwidthPerSec’
have any notable effect on apparent balancer rate.  This is on Hadoop 2.2.0

john

From: cho ju il [mailto:tjstory@kgrid.co.kr]
Sent: Wednesday, September 03, 2014 12:55 AM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: RE: How can I increase the speed balancing?


Bandwidth is enough

And i use command "bin/hdfs dfsadmin -setBalancerBandwidth 52428800 ".

Yet balancing slow.

I think because the file transfer speed is slow and  move 5 files per 1 server .

-----Original Message-----
From: "Srikanth upputuri"<srikanth.upputuri@huawei.com<mailto:srikanth.upputuri@huawei.com>>
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"<user@hadoop.apache.org<mailto:user@hadoop.apache.org>>;
Cc:
Sent: 2014-09-03 (수) 14:10:24
Subject: RE: How can I increase the speed balancing?

I am not sure what you meant by ‘Bandwidth is not a lack of data nodes’ but have you configured
the balancer bandwidth property ‘dfs.datanode.balance.bandwidthPerSec’? If not it defaults
to 1KB/s. You can increase this to improve the balancer speed. You may also set it dynamically
using the command ‘dfsadmin -setBalanacerBandwidth newbandwidth’ before running balancer.





From: cho ju il [mailto:tjstory@kgrid.co.kr]
Sent: 03 September 2014 06:31
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: How can I increase the speed balancing?



hadoop version 2.4.1

Balancing speed is slow.

I think because the file transfer speed is slow.

Bandwidth is not a lack of data nodes.

How can I increase the speed balancing?









*** balancer log

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.207:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.205:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.203:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.210:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.211:40010

2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.114:40010

2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.206:40010

2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.201:40010

2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.202:40010

2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.204:40010

2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 over-utilized:
[Source[192.168.0.203:40010, utilization=99.99693907952093], Source[192.168.0.201:40010, utilization=99.99713240471648],
Source[192.168.0.202:40010, utilization=99.99652052169367]]

2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 underutilized:
[BalancerDatanode[192.168.0.211:40010, utilization=62.735024524531006], BalancerDatanode[192.168.0.114:40010,
utilization=2.3174560700459224]]

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move
70.30 TB to make the cluster balanced.

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move
10 GB bytes from 192.168.0.203:40010 to 192.168.0.211:40010

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move
10 GB bytes from 192.168.0.201:40010 to 192.168.0.114:40010

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 20
GB in this iteration

2014-09-03 09:44:23,643 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216256_3475617 with size=1746577 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.205:40010

2014-09-03 09:45:38,730 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1076170742_2430103 with size=16746627 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.203:40010

2014-09-03 09:46:42,649 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216315_3475676 with size=38338949 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.206:40010

2014-09-03 09:50:13,239 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1074229029_488207 with size=89080307 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.203:40010

2014-09-03 09:50:58,659 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216317_3475678 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.210:40010

2014-09-03 09:51:15,660 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1075275002_1534363 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.204:40010

2014-09-03 09:51:19,661 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216257_3475618 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010
through 192.168.0.202:40010

2014-09-03 09:51:54,243 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1076170740_2430101 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.203:40010

2014-09-03 09:52:15,242 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1074229048_488226 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.205:40010

2014-09-03 09:52:22,744 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1076170741_2430102 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010
through 192.168.0.203:40010



[Image removed by sender.]



[Image removed by sender.]


Mime
View raw message