hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
Date Thu, 17 Sep 2015 17:14:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803246#comment-14803246
] 

Brahma Reddy Battula commented on HDFS-8647:
--------------------------------------------

{{TestBlocksWithNotEnoughRacks}} are failing due to following cases.

 *Before change* 

 if cluster was multi rack first and then later became single rack, then those blocks were
getting added in {{NeededReplications}}, and the same value was expected in tests. 
   a. But this was only till the namenode was alive. If NN was restarted after making into
single rack, this {{NeededReplications}} will not have the block.
   b. And before NN restart, another rack was added, then auto replication happens to new
rack. But Once NN restarted, and new rack added, then auto replication to new rack (if single
rack have already enough replicas == RF) to new rack happens only if RF changes on those blocks.

 *After change*

it will not add to {{NeededReplications}} immediately cluster became the single rack.
  a. After change, auto replication (if single rack have already enough replicas == RF)  will
not happen when the cluster is added with one more rack. Only this will be triggered only
if RF changes on those blocks.

If new change is Okay, then test case can be updated, else patch can be updated.

> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -------------------------------------------------------------
>
>                 Key: HDFS-8647
>                 URL: https://issues.apache.org/jira/browse/HDFS-8647
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch, HDFS-8647-003.patch
>
>
> Sometimes we want to have namenode use alternative block placement policy such as upgrade
domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as useDelHint,
blockHasEnoughRacks. That means when we have new block placement policy, we need to modify
BlockManager to account for the new policy. Ideally BlockManager should ask BlockPlacementPolicy
object instead. That will allow us to provide new BlockPlacementPolicy without changing BlockManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message