hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8884) Fail-fast check in BlockPlacementPolicyDefault#chooseTarget
Date Tue, 18 Aug 2015 11:51:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701127#comment-14701127
] 

Vinayakumar B commented on HDFS-8884:
-------------------------------------

Patch looks overall good. Thanks [~hitliuyi]

1. There are some nits, as per checkstyle, needs to be cleaned.

2. In test, {{testPlacementWithLocalRackNodesDecommissioned}} doesn't ensure that {{dnd3}}
belongs to client's rack. Add a check before verifying placement.

+1 once addressed.


> Fail-fast check in BlockPlacementPolicyDefault#chooseTarget
> -----------------------------------------------------------
>
>                 Key: HDFS-8884
>                 URL: https://issues.apache.org/jira/browse/HDFS-8884
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>         Attachments: HDFS-8884.001.patch
>
>
> In current BlockPlacementPolicyDefault, when choosing datanode storage to place block,
we have following logic:
> {code}
>         final DatanodeStorageInfo[] storages = DFSUtil.shuffle(
>             chosenNode.getStorageInfos());
>         int i = 0;
>         boolean search = true;
>         for (Iterator<Map.Entry<StorageType, Integer>> iter = storageTypes
>             .entrySet().iterator(); search && iter.hasNext(); ) {
>           Map.Entry<StorageType, Integer> entry = iter.next();
>           for (i = 0; i < storages.length; i++) {
>             StorageType type = entry.getKey();
>             final int newExcludedNodes = addIfIsGoodTarget(storages[i],
> {code}
> We will iterate (actually two {{for}}, although they are usually small value) all storages
of the candidate datanode even the datanode itself is not good (e.g. decommissioned, stale,
too busy..), since currently we do all the check in {{addIfIsGoodTarget}}.
> We can fail-fast: check the datanode related conditions first, if the datanode is not
good, then no need to shuffle and iterate the storages. Then it's more efficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message