hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1547) Improve decommission mechanism
Date Wed, 12 Jan 2011 21:08:50 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980930#action_12980930
] 

Todd Lipcon commented on HDFS-1547:
-----------------------------------

The issue with isGoodTarget() is that it doesn't just prioritize low-load nodes, it actively
excludes any node with load > 2*avgLoad. Consider this situation:

- 10 nodes, each of which has 2 transceivers active (total load = 20)
- decommission 6 of them
- total load is now only 8 (from the remaining ones) but "size" is still 10
- avgLoad = 0.8
- any node with load >1.6 will not be chosen
- thus all nodes are excluded and we cannot allocate a block

That is to say, decomissioning nodes now contribute to the denominator of the total/numNodes
fraction but not the numerator, so it drives the average down.

Does this make sense?

> Improve decommission mechanism
> ------------------------------
>
>                 Key: HDFS-1547
>                 URL: https://issues.apache.org/jira/browse/HDFS-1547
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1547.1.patch, HDFS-1547.patch
>
>
> Current decommission mechanism driven using exclude file has several issues. This bug
proposes some changes in the mechanism for better manageability. See the proposal in the next
comment for more details.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message