hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashish Singhi (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3286) When the threshold value for balancer is 0(zero) ,unexpected output is displayed
Date Wed, 18 Apr 2012 15:40:44 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256649#comment-13256649
] 

Ashish Singhi commented on HDFS-3286:
-------------------------------------

@Uma: Yes you are right.
I tried this by the following code modification,
{code}
-      if (utilization >= avgUtil+threshold
-          || utilization <= avgUtil-threshold) { 
-        maxSizeToMove = (long)(threshold*datanode.getCapacity()/100);
+      if (utilization >= avgUtil + threshold
+          || utilization <= avgUtil - threshold) {
+        if (threshold != 0){
+          maxSizeToMove = (long) (threshold * datanode.getCapacity() / 100);
+        } else {
+          maxSizeToMove = (long) (datanode.getCapacity() / 100);
+        }
       } else {
{code}

Is this modification of code, ok ?
But with 0 threshold, balancer keeps iterating because it is very difficult to produce a threshold
of 0% between the datanodes.

I wrote a test case, cluster with 1DN then filled it 30% and then starting one more say DN2.
This will take very large to get terminate.
One possible way would be to have a cluster with 1DN fill it 30% and then starting one more
say DN2 with the same usage as DN1. Then it will terminate within very less time with ReturnStatus.SUCCESS.
Is it ok to have a test case like that ?
                
> When the threshold value for balancer is 0(zero) ,unexpected output is displayed
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-3286
>                 URL: https://issues.apache.org/jira/browse/HDFS-3286
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer
>    Affects Versions: 0.23.0
>            Reporter: J.Andreina
>             Fix For: 0.24.0
>
>
> Replication factor =1
> Step 1: Start NN,DN1.write 4 GB of data
> Step 2: Start DN2
> Step 3: issue the balancer command(./hdfs balancer -threshold 0)
> The threshold parameter is a fraction in the range of (0%, 100%) with a default value
of 10%
> When the above scenario is executed the Source DN and Target DN is choosen and the number
of bytes to be moved from source to target DN is also calculated .
> Then the balancer is exiting with the following message "No block can be moved. Exiting..."
which is not expected.
> {noformat}
> HOST-xx-xx-xx-xx:/home/Andreina/APril10/install/hadoop/namenode/bin # ./hdfs balancer
-threshold 0
> 12/04/16 16:22:07 INFO balancer.Balancer: Using a threshold of 0.0
> 12/04/16 16:22:07 INFO balancer.Balancer: namenodes = [hdfs://HOST-xx-xx-xx-xx:9000]
> 12/04/16 16:22:07 INFO balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node,
threshold=0.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being
Moved
> 12/04/16 16:22:10 INFO net.NetworkTopology: Adding a new node: /default-rack/yy.yy.yy.yy:50176
> 12/04/16 16:22:10 INFO net.NetworkTopology: Adding a new node: /default-rack/xx.xx.xx.xx:50010
> 12/04/16 16:22:10 INFO balancer.Balancer: 1 over-utilized: [Source[xx.xx.xx.xx:50010,
utilization=7.212458091389678]]
> 12/04/16 16:22:10 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[yy.yy.yy.yy:50176,
utilization=4.650670324367203E-5]]
> 12/04/16 16:22:10 INFO balancer.Balancer: Need to move 1.77 GB to make the cluster balanced.
> No block can be moved. Exiting...
> Balancing took 5.142 seconds
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message