hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
Date Wed, 16 Sep 2015 16:41:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790695#comment-14790695

Brahma Reddy Battula commented on HDFS-8647:

{quote}Upgrading from 2.7 to 2.8 (assume we put this change in 2.8) will break that unless
they recompile their custom class. That is because BlockManager will start calling {{chooseReplicasToDelete}}
method which isn't implemented in their class.{quote}

AFAIK most customized policy extends BlockPlacementPolicyDefault. It's rare to implement a
policy from scratch. Not only method signature, as long as we change default policy one byte,
the customized policy need recompile anyway. Maven can solve the dependency and recompile
if needed. 
The old {{chooseReplicasToDelete}} is limited. I suppose little user override it. After the
change, it's more flexible to let user customize it. If user didn't override it before, then
the customized policy class can cleanly recompile without complain.

{quote}Note that this seems similar to BlockPlacementPolicy's verifyBlockPlacement. So if
they can be combined, that is nice.{quote}
That's a good idea.

> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -------------------------------------------------------------
>                 Key: HDFS-8647
>                 URL: https://issues.apache.org/jira/browse/HDFS-8647
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch, HDFS-8647-003.patch
> Sometimes we want to have namenode use alternative block placement policy such as upgrade
domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as useDelHint,
blockHasEnoughRacks. That means when we have new block placement policy, we need to modify
BlockManager to account for the new policy. Ideally BlockManager should ask BlockPlacementPolicy
object instead. That will allow us to provide new BlockPlacementPolicy without changing BlockManager.

This message was sent by Atlassian JIRA

View raw message