hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3495) Update Balancer to support new NetworkTopology with NodeGroup
Date Mon, 26 Nov 2012 20:50:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504074#comment-13504074
] 

Tsz Wo (Nicholas), SZE commented on HDFS-3495:
----------------------------------------------

Patch looks good.  Just some minor comments:
- In Balancer.chooseProxySource(), "proxySource = loc" is not needed since addTo(..) already
has set it.  Also, the indentation for the first if-statement has 4 spaces.

- TestBalancerWithNodeGroup has some javac warnings
-* Random r is not used.
-* unused imports (your may use [ctrl/cmd]-shift-o to fix it in eclipse)

- There are also unused imports in Balancer.

- Typo: "paramater" => "parameter" in NetworkTopology.getInstance.

- Change the new comments to javadoc whenever possible, e.g. use /** insteadof /* for method
header comments.

                
> Update Balancer to support new NetworkTopology with NodeGroup
> -------------------------------------------------------------
>
>                 Key: HDFS-3495
>                 URL: https://issues.apache.org/jira/browse/HDFS-3495
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer
>    Affects Versions: 1.1.0, 2.0.2-alpha
>            Reporter: Junping Du
>            Assignee: Junping Du
>         Attachments: HADOOP-8473-Balancer-NodeGroup-aware.patch, HDFS-3495-v2.patch,
HDFS-3495-v3.patch, HDFS-3495-v4.patch
>
>
> Since the Balancer is a Hadoop Tool, it was updated to be directly aware of four-layer
hierarchy instead of creating an alternative Balancer implementation. To accommodate extensibility,
a new protected method, doChooseNodesForCustomFaultDomain is now called from the existing
chooseNodes method so that a subclass of the Balancer could customize the balancer algotirhm
for other failure and locality topologies. An alternative option is to encapsulate the algorithm
used for the four-layer hierarchy into a collaborating strategy class.
> The key changes introduced to support a four-layer hierarchy were to override the algorithm
of choosing <source, target> pairs for balancing. Unit tests were created to test the
new algorithm.
> The algorithm now makes sure to choose the target and source node on the same node group
for balancing as the first priority. Then the overall balancing policy is: first doing balancing
between nodes within the same nodegroup then the same rack and off rack at last. Also, we
need to check no duplicated replicas live in the same node group after balancing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message