hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8884) Fail-fast check in BlockPlacementPolicyDefault#chooseTarget
Date Thu, 20 Aug 2015 12:16:47 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yi Liu updated HDFS-8884:
-------------------------
       Resolution: Fixed
     Hadoop Flags: Reviewed
    Fix Version/s: 2.8.0
           Status: Resolved  (was: Patch Available)

> Fail-fast check in BlockPlacementPolicyDefault#chooseTarget
> -----------------------------------------------------------
>
>                 Key: HDFS-8884
>                 URL: https://issues.apache.org/jira/browse/HDFS-8884
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>             Fix For: 2.8.0
>
>         Attachments: HDFS-8884.001.patch, HDFS-8884.002.patch
>
>
> In current BlockPlacementPolicyDefault, when choosing datanode storage to place block,
we have following logic:
> {code}
>         final DatanodeStorageInfo[] storages = DFSUtil.shuffle(
>             chosenNode.getStorageInfos());
>         int i = 0;
>         boolean search = true;
>         for (Iterator<Map.Entry<StorageType, Integer>> iter = storageTypes
>             .entrySet().iterator(); search && iter.hasNext(); ) {
>           Map.Entry<StorageType, Integer> entry = iter.next();
>           for (i = 0; i < storages.length; i++) {
>             StorageType type = entry.getKey();
>             final int newExcludedNodes = addIfIsGoodTarget(storages[i],
> {code}
> We will iterate (actually two {{for}}, although they are usually small value) all storages
of the candidate datanode even the datanode itself is not good (e.g. decommissioned, stale,
too busy..), since currently we do all the check in {{addIfIsGoodTarget}}.
> We can fail-fast: check the datanode related conditions first, if the datanode is not
good, then no need to shuffle and iterate the storages. Then it's more efficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message