hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cho ju il <tjst...@kgrid.co.kr>
Subject RE: How can I increase the speed balancing?
Date Wed, 03 Sep 2014 06:54:52 GMT
Bandwidth is enough 
And i use command "bin/hdfs dfsadmin -setBalancerBandwidth 52428800 ".
Yet balancing slow.
I think because the file transfer speed is slow and  move 5 files per 1 server .




 
 
 
 
 
-----Original Message-----
From: "Srikanth upputuri"&lt;srikanth.upputuri@huawei.com&gt; 
To: "user@hadoop.apache.org"&lt;user@hadoop.apache.org&gt;; 
Cc: 
Sent: 2014-09-03 (수) 14:10:24
Subject: RE: How can I increase the speed balancing?










I am not sure what you meant by ‘Bandwidth is not a lack of data nodes’ but have you configured
the balancer bandwidth property ‘dfs.datanode.balance.bandwidthPerSec’? If not it defaults
to 1KB/s. You can increase this to improve the
 balancer speed. You may also set it dynamically using the command ‘dfsadmin -setBalanacerBandwidth
newbandwidth’ before running balancer.

 

 


From: cho ju il [mailto:tjstory@kgrid.co.kr]


Sent: 03 September 2014 06:31

To: user@hadoop.apache.org

Subject: How can I increase the speed balancing?


 


hadoop version 2.4.1

Balancing speed is slow.

I think because the file transfer speed is slow. 

Bandwidth is not a lack of data nodes. 

How can I increase the speed balancing?

 

 

 

 

*** balancer log

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.207:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.205:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.203:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.210:40010

2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.211:40010

2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.114:40010

2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.206:40010

2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.201:40010

2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.202:40010

2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.204:40010

2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 over-utilized:
[Source[192.168.0.203:40010, utilization=99.99693907952093], Source[192.168.0.201:40010, utilization=99.99713240471648],
 Source[192.168.0.202:40010, utilization=99.99652052169367]]

2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 underutilized:
[BalancerDatanode[192.168.0.211:40010, utilization=62.735024524531006], BalancerDatanode[192.168.0.114:40010,
 utilization=2.3174560700459224]]

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move
70.30 TB to make the cluster balanced.

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move
10 GB bytes from 192.168.0.203:40010 to 192.168.0.211:40010

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move
10 GB bytes from 192.168.0.201:40010 to 192.168.0.114:40010

2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 20
GB in this iteration

2014-09-03 09:44:23,643 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216256_3475617 with size=1746577 from 192.168.0.201:40010
 to 192.168.0.114:40010 through 192.168.0.205:40010

2014-09-03 09:45:38,730 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1076170742_2430103 with size=16746627 from 192.168.0.203:40010 to 192.168.0.211:40010
through
 192.168.0.203:40010

2014-09-03 09:46:42,649 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216315_3475676 with size=38338949 from 192.168.0.201:40010 to 192.168.0.114:40010
through
 192.168.0.206:40010

2014-09-03 09:50:13,239 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1074229029_488207 with size=89080307 from 192.168.0.203:40010 to 192.168.0.211:40010
through
 192.168.0.203:40010

2014-09-03 09:50:58,659 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216317_3475678 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010
through
 192.168.0.210:40010

2014-09-03 09:51:15,660 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1075275002_1534363 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010
through
 192.168.0.204:40010

2014-09-03 09:51:19,661 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1077216257_3475618 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010
through
 192.168.0.202:40010

2014-09-03 09:51:54,243 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1076170740_2430101 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010
through
 192.168.0.203:40010

2014-09-03 09:52:15,242 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1074229048_488226 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010
through
 192.168.0.205:40010

2014-09-03 09:52:22,744 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully
moved blk_1076170741_2430102 with size=134217728 from 192.168.0.203:40010
 to 192.168.0.211:40010 through 192.168.0.203:40010

 








 






Mime
View raw message