hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8884) Improve BlockPlacementPolicyDefault#chooseTarget to check candidate datanode first
Date Tue, 11 Aug 2015 07:36:46 GMT
Yi Liu created HDFS-8884:

             Summary: Improve BlockPlacementPolicyDefault#chooseTarget to check candidate
datanode first 
                 Key: HDFS-8884
                 URL: https://issues.apache.org/jira/browse/HDFS-8884
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Yi Liu
            Assignee: Yi Liu

In current BlockPlacementPolicyDefault, when choosing datanode storage to place block, we
have following logic:

        final DatanodeStorageInfo[] storages = DFSUtil.shuffle(
        int i = 0;
        boolean search = true;
        for (Iterator<Map.Entry<StorageType, Integer>> iter = storageTypes
            .entrySet().iterator(); search && iter.hasNext(); ) {
          Map.Entry<StorageType, Integer> entry = iter.next();
          for (i = 0; i < storages.length; i++) {
            StorageType type = entry.getKey();
            final int newExcludedNodes = addIfIsGoodTarget(storages[i],
We will iterate (actually two {{for}}) all storages of the candidate datanode even the datanode
itself is not good (e.g. decommissioned, stale, too busy..), since currently we do all the
check in {{addIfIsGoodTarget}}.

We can fail-fast: check the datanode related conditions first, if the datanode is not good,
then no need to shuffle and iterate the storages. Then it's more efficient.

This message was sent by Atlassian JIRA

View raw message