hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-972) Improve the rack-aware replica placement performance
Date Fri, 16 Feb 2007 01:01:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473576

Hairong Kuang commented on HADOOP-972:

I agree that ReplicaChooser is a better name. But I am not sure if the class should be moved
out of FSNamesystem. In additon to clusterMap, it also needs to calculate the avgLoad per
data node. FSNameSystem provides the total load of the cluster.

> Improve the rack-aware replica placement performance
> ----------------------------------------------------
>                 Key: HADOOP-972
>                 URL: https://issues.apache.org/jira/browse/HADOOP-972
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.11.0
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>             Fix For: 0.12.0
>         Attachments: rack_performance.patch
> This issue aims to improve the rack-aware replica placement performance. A major idea
is to avoid constructing lists of possible targets for random selection in chooseTarget, which
currently needs interating all DatanodeDescriptors. I plan to change the NetworkTopology data
structure as follow:
> 1. each InnerNode stores its childrens as a list;
> 2. each InnerNode adds a new field numberOfLeaves the total number of leaves (i.e. data
nodes) in its subtree. 
> NetworkTopology will support two new methods:
> 1. DatanodeDescriptor chooseRandom( String scope): it randomly choose one leave from
> 2. DatanodeDescriptor chooseRandomExclude(String excludedScope): it randomly choose one
leave from ~scope
> In addition, Issue 971 will also help improve the performance of the rack-aware DFS patch.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message