hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
Date Wed, 14 Oct 2015 16:48:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957248#comment-14957248

Ming Ma commented on HDFS-8647:

Thanks [~vinayrpet]!

bq. how about waiting for commit to branch-2. After that simple cherry-pick might work.
I was trying to point out that cherry-pick likely won't work. It is just to make sure after
I commit it to trunk, there is patch for reference in case I need to manually resolve it as
part of cherry-pick.

bq. "# of racks >= # of data blocks" requirement is to ensure rackwise failure doesn't
create any dataloss in case of EC'ed file.
Is that to ensure no data loss or more for read reconstruction time optimization? Use the
example of RS(6,3) and 5 racks, the blocks will be written successfully. If rack_0 fails,
read needs to reconstruct data_0 and data_5, but there is no data loss. This is just for my
own education, to clarify why the write policy is different from verification policy for EC
scenario. Again, this is not related this jira.


> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -------------------------------------------------------------
>                 Key: HDFS-8647
>                 URL: https://issues.apache.org/jira/browse/HDFS-8647
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch, HDFS-8647-003.patch, HDFS-8647-004.patch,
HDFS-8647-004.patch, HDFS-8647-005.patch
> Sometimes we want to have namenode use alternative block placement policy such as upgrade
domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as useDelHint,
blockHasEnoughRacks. That means when we have new block placement policy, we need to modify
BlockManager to account for the new policy. Ideally BlockManager should ask BlockPlacementPolicy
object instead. That will allow us to provide new BlockPlacementPolicy without changing BlockManager.

This message was sent by Atlassian JIRA

View raw message