hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Butler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2767) org.apache.hadoop.net.NetworkTopology.InnerNode#getLeaf does not return the last node on a rack when used with an excluded node
Date Fri, 01 Feb 2008 18:27:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564858#action_12564858
] 

Mark Butler commented on HADOOP-2767:
-------------------------------------

Further examination shows the problem only occurs when you are excluding a node, not when
you are excluding a rack. So it may not be a problem for DFS ...

However, if it does occur, it is not just a problem of not using a node, it also causes another
node to be a "hotspot" as it is twice as likely as any other node to get a replica placement.,
which is clearly undesirable ..  

Can someone more familiar with DFS replication code comment if this is likely to be a problem?



> org.apache.hadoop.net.NetworkTopology.InnerNode#getLeaf does not return the last node
on a rack when used with an excluded node
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2767
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2767
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: hudson
>            Reporter: Mark Butler
>            Priority: Minor
>             Fix For: hudson
>
>         Attachments: NetworkTopologyTest.java
>
>
> I have written some test code that shows NetworkTopology.InnerNode#getLeaf will never
return the last node on the rack if it is called with an excludedNode (for example the first
node on the rack). 
> Consequently I suspect that NetworkTopology.chooseRandom() will never returns the last
node on the remote rack for the second replica in DFS. 
> I have some test code that demonstrates this problem at the getLeaf level, although it
is necessary to change the visibility of the NetworkTopology.InnerNode, NetworkTopology.InnerNode#getLeaf
and NetworkTopology.getNode from private to package default to run the test. 
> TODO: Demonstrate problem at NetworkTopology.chooseRandom level, then submit candidate
fix. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message