hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
Date Sun, 18 Oct 2015 23:58:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14962697#comment-14962697
] 

Ming Ma commented on HDFS-8647:
-------------------------------

Most of the test failures are due to mismatch between BlockManager and function definition
of {{chooseReplicasToDelete}}; BlockManager should swap {{addedNode}} and {{delNodeHint}}.

After the fix, the only failed unit test left is {{TestBalancer#testBalancerWithPinnedBlocks}}.
Can you please investigate if it is related to the patch?

Nits:
* {{public}} can be removed from {{public DatanodeStorageInfo chooseReplicaToDelete}}.
* {{final List<DatanodeStorageInfo> moreThanOne = new ArrayList<DatanodeStorageInfo>();}}
Given hadoop uses JDK7, you can simplify this with type inference {{final List<DatanodeStorageInfo>
moreThanOne = new ArrayList<>();}} Same for other places in the patch.
* {{chooseReplicaToDelete}}’s parameter {{Collection<DatanodeStorageInfo> first}}.
How about name it {{moreThanOne}} instead? Similarly for {{second}}.


> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -------------------------------------------------------------
>
>                 Key: HDFS-8647
>                 URL: https://issues.apache.org/jira/browse/HDFS-8647
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch, HDFS-8647-003.patch, HDFS-8647-004.patch,
HDFS-8647-004.patch, HDFS-8647-005.patch, HDFS-8647-006.patch
>
>
> Sometimes we want to have namenode use alternative block placement policy such as upgrade
domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as useDelHint,
blockHasEnoughRacks. That means when we have new block placement policy, we need to modify
BlockManager to account for the new policy. Ideally BlockManager should ask BlockPlacementPolicy
object instead. That will allow us to provide new BlockPlacementPolicy without changing BlockManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message