hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
Date Fri, 25 Sep 2015 23:59:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908908#comment-14908908

Ming Ma commented on HDFS-8647:

Thanks [~brahmareddy]. Maybe we can move {{hasClusterEverBeenMultiRack}} from DatanodeManager
to NetworkTopology? Then {{BlockPlacementPolicyDefault}}'s {{verifyBlockPlacement}} can ask
{{clusterMap}} if the cluster has ever been multi rack. In that way, we completely remove
the multi rack reference from BlockManager.

Regardless the approach, there is a behavior change for {{BlockPlacementPolicyDefault}}'s
{{verifyBlockPlacement}}, which is used by fsck. When # of racks is reduced to 1, fsck used
to return ok; but with the change, it will indicate it violates the rack policy. That should
be ok.

Nits: could you please clean up the whitespace? Also the descriptions you added {{chooseReplicaToDelete}}
don't match the parameter names.

> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -------------------------------------------------------------
>                 Key: HDFS-8647
>                 URL: https://issues.apache.org/jira/browse/HDFS-8647
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch, HDFS-8647-003.patch, HDFS-8647-004.patch
> Sometimes we want to have namenode use alternative block placement policy such as upgrade
domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as useDelHint,
blockHasEnoughRacks. That means when we have new block placement policy, we need to modify
BlockManager to account for the new policy. Ideally BlockManager should ask BlockPlacementPolicy
object instead. That will allow us to provide new BlockPlacementPolicy without changing BlockManager.

This message was sent by Atlassian JIRA

View raw message