hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-972) Improve the rack-aware replica placement performance
Date Mon, 05 Feb 2007 02:22:05 GMT
Improve the rack-aware replica placement performance
----------------------------------------------------

                 Key: HADOOP-972
                 URL: https://issues.apache.org/jira/browse/HADOOP-972
             Project: Hadoop
          Issue Type: Improvement
          Components: dfs
    Affects Versions: 0.11.0
            Reporter: Hairong Kuang
         Assigned To: Hairong Kuang
             Fix For: 0.12.0


This issue aims to improve the rack-aware replica placement performance. A major idea is to
avoid constructing lists of possible targets for random selection in chooseTarget, which currently
needs interating all DatanodeDescriptors. I plan to change the NetworkTopology data structure
as follow:
1. each InnerNode stores its childrens as a list;
2. each InnerNode adds a new field numberOfLeaves the total number of leaves (i.e. data nodes)
in its subtree. 
NetworkTopology will support two new methods:
1. DatanodeDescriptor chooseRandom( String scope): it randomly choose one leave from scope.
2. DatanodeDescriptor chooseRandomExclude(String excludedScope): it randomly choose one leave
from ~scope

In addition, Issue 971 will also help improve the performance of the rack-aware DFS patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message