hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashish Singhi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3377) While Balancing more than 10 Blocks are being moved from one DN even though the maximum number of blocks to be moved in an iterations is hard coded to 5
Date Mon, 04 Jun 2012 14:46:23 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288601#comment-13288601
] 

Ashish Singhi commented on HDFS-3377:
-------------------------------------

Hi Andreina.

There is no restriction on the number of blocks a datanode can move in one iteration. 

{quote}But MAX_NUM_CONCURRENT_MOVES in one iterations is hard coded to 5.{quote} 

MAX_NUM_CONCURRENT_MOVES - specifies the maximum number of concurrent block moves for balancing
purpose at a datanode can be scheduled at one single point of time(range varies from cluster
to cluster), not in one iteration. 

Following are the balancer outputs obtained by executing the adove mentioned scenario. 
{code}
./hdfs balancer -threshold 1 
Listening for transport dt_socket at address: 8050 
12/06/04 11:38:47 INFO balancer.Balancer: Using a threshold of 1.0 
12/06/04 11:38:47 INFO balancer.Balancer: namenodes = [hdfs://HOST-10-18-40-117:9000] 
12/06/04 11:38:47 INFO balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node,
threshold=1.0] 
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being
Moved 
12/06/04 11:38:51 INFO balancer.Balancer: Block token params received from NN: keyUpdateInterval=600
min(s), tokenLifetime=600 min(s) 
12/06/04 11:38:52 INFO block.BlockTokenSecretManager: Setting block keys 
12/06/04 11:38:52 INFO balancer.Balancer: Balancer will update its block keys every 150 minute(s)

12/06/04 11:38:52 INFO block.BlockTokenSecretManager: Setting block keys 
12/06/04 11:38:52 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/xx.xx.xx.xx:50276

12/06/04 11:38:52 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/xx.xx.xx.xx:50176

12/06/04 11:38:52 INFO balancer.Balancer: 1 over-utilized: [Source[xx.xx.xx.xx:50176, utilization=8.37380402300465]]

12/06/04 11:38:52 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[xx.xx.xx.xx:50276,
utilization=2.8947382057801816]] 
12/06/04 11:38:52 INFO balancer.Balancer: Need to move 1.22 GB to make the cluster balanced.

12/06/04 11:38:52 INFO balancer.Balancer: Decided to move 716.13 MB bytes from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 
12/06/04 11:38:52 INFO balancer.Balancer: Will move 716.13 MB in this iteration 
May 18, 2012 11:38:52 AM          0                 0 KB             1.22 GB          716.13
MB 
12/06/04 11:44:06 INFO balancer.Balancer: Moving block -604085946676187269 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:44:11 INFO balancer.Balancer: Moving block -8407984470383132708 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:44:28 INFO balancer.Balancer: Moving block -6889513481040935685 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:44:45 INFO balancer.Balancer: Moving block 8195882447563616912 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:44:51 INFO balancer.Balancer: Moving block 8785624464350544567 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:46:55 INFO balancer.Balancer: Moving block 2826762635631555809 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:49:55 INFO balancer.Balancer: Moving block -8045953083201474660 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:50:24 INFO balancer.Balancer: Moving block 7717431073633882015 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:50:27 INFO balancer.Balancer: Moving block 6479080974771358747 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:51:15 INFO balancer.Balancer: Moving block -8899955335678387087 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:51:49 INFO balancer.Balancer: Moving block 1070038993680508333 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:52:02 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/xx.xx.xx.xx:50176

12/06/04 11:52:02 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/xx.xx.xx.xx:50276

12/06/04 11:52:02 INFO balancer.Balancer: 1 over-utilized: [Source[xx.xx.xx.xx:50176, utilization=7.332913449391756]]

12/06/04 11:52:02 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[xx.xx.xx.xx:50276,
utilization=4.231341505676603]] 
12/06/04 11:52:02 INFO balancer.Balancer: Need to move 394.43 MB to make the cluster balanced.

12/06/04 11:52:02 INFO balancer.Balancer: Decided to move 716.13 MB bytes from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 
12/06/04 11:52:02 INFO balancer.Balancer: Will move 716.13 MB in this iteration 
May 18, 2012 11:52:02 AM          1                 0 KB           394.43 MB          716.13
MB 
12/06/04 11:57:25 INFO balancer.Balancer: Moving block -5591766437649317399 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:57:42 INFO balancer.Balancer: Moving block 9198575182337433559 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:57:56 INFO balancer.Balancer: Moving block 6855524568701374882 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:58:02 INFO balancer.Balancer: Moving block 4747385924662820878 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 11:58:29 INFO balancer.Balancer: Moving block -5973238543364868278 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 12:02:58 INFO balancer.Balancer: Moving block 5717753712470906112 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 12:03:06 INFO balancer.Balancer: Moving block -3396403038672813493 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 12:03:42 INFO balancer.Balancer: Moving block 175510590632581188 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 12:03:48 INFO balancer.Balancer: Moving block 735609460427544500 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 12:04:01 INFO balancer.Balancer: Moving block 9011172593322217078 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 12:05:42 INFO balancer.Balancer: Moving block 3659606766577555250 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 12:05:44 INFO balancer.Balancer: Moving block 3186378020600152947 from xx.xx.xx.xx:50176
to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded. 
12/06/04 12:06:12 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/xx.xx.xx.xx:50176

12/06/04 12:06:12 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/xx.xx.xx.xx:50276

12/06/04 12:06:12 INFO balancer.Balancer: 0 over-utilized: [] 
12/06/04 12:06:12 INFO balancer.Balancer: 0 underutilized: [] 
The cluster is balanced. Exiting... 
12/06/04 12:06:13 INFO balancer.Balancer: InterruptedException in block key updater thread

java.lang.InterruptedException: sleep interrupted 
        at java.lang.Thread.sleep(Native Method) 
        at org.apache.hadoop.hdfs.server.balancer.NameNodeConnector$BlockKeyUpdater.run(NameNodeConnector.java:200)

        at java.lang.Thread.run(Thread.java:619) 
Balancing took 27.499833333333335 minutes 
{code}
Here you can observe that, 
In the first iteration, the first 5 blocks had moved between 11:44:06 - 11:44:51 
and the next 5 blocks had been moved between 11:49:55 - 11:51:49. 
So at one single point of time (here the range is one minute) there are no more than 5 blocks
movement from the datanode. 

Conclusion - Balancer can move more than 5 blocks from one datanode in one iteration but not
at a single point of time. 

                
> While Balancing more than 10 Blocks are being moved from one DN even though the maximum
number of blocks to be moved in an iterations is hard coded to 5
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3377
>                 URL: https://issues.apache.org/jira/browse/HDFS-3377
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer
>    Affects Versions: 2.0.0-alpha
>            Reporter: J.Andreina
>
> Replication factor= 1,block size is default value
> Step 1: Start NN,DN1
> Step 2: Pump 5 GB of data.
> Step 3: Start DN2 and issue balancer with threshold value 1
> In the balancer report and the NN logs displays that more than 8 blocks are being moved
from DN1 to DN2 in one iterations But MAX_NUM_CONCURRENT_MOVES in one iterations is hard coded
to 5.
> Balancer report for 1st iteration:
> =================================
> {noformat}
> HOST-XX-XX-XX-XX:/home/Andreina/NewHadoop2nd/hadoop-2.0.0-SNAPSHOT/bin # ./hdfs balancer
-threshold 1
> 12/05/03 17:31:28 INFO balancer.Balancer: Using a threshold of 1.0
> 12/05/03 17:31:28 INFO balancer.Balancer: namenodes = [hdfs://HOST-XX-XX-XX-XX:9002]
> 12/05/03 17:31:28 INFO balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node,
threshold=1.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being
Moved
> 12/05/03 17:31:30 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/YY.YY.YY.YY:50176
> 12/05/03 17:31:30 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/XX.XX.XX.XX:50076
> 12/05/03 17:31:30 INFO balancer.Balancer: 1 over-utilized: [Source[XX.XX.XX.XX:50076,
utilization=5.018416429773605]]
> 12/05/03 17:31:30 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[YY.YY.YY.YY:50176,
utilization=3.272819804269012E-5]]
> 12/05/03 17:31:30 INFO balancer.Balancer: Need to move 1.06 GB to make the cluster balanced.
> 12/05/03 17:31:30 INFO balancer.Balancer: Decided to move 716.13 MB bytes from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176
> 12/05/03 17:31:30 INFO balancer.Balancer: Will move 716.13 MB in this iteration
> May 3, 2012 5:31:30 PM            0                 0 KB             1.06 GB        
 716.13 MB
> 12/05/03 17:35:29 INFO balancer.Balancer: Moving block -5275260117334749945 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:36:31 INFO balancer.Balancer: Moving block -8079758341763366944 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:37:12 INFO balancer.Balancer: Moving block -7395554712490186313 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:37:45 INFO balancer.Balancer: Moving block 7805443002654525130 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:38:15 INFO balancer.Balancer: Moving block 1864290085256894184 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:40:30 INFO balancer.Balancer: Moving block 23322655230037442 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:41:24 INFO balancer.Balancer: Moving block -8839566903692469634 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:43:03 INFO balancer.Balancer: Moving block 7304385435779271887 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:43:48 INFO balancer.Balancer: Moving block -7242009026552182303 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:44:06 INFO balancer.Balancer: Moving block -2449309138254106767 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:44:55 INFO balancer.Balancer: Moving block 500930296233438046 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:45:04 INFO balancer.Balancer: Moving block 2642725820310610865 from XX.XX.XX.XX:50076
to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message