hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manoj Govindassamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10530) BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy
Date Sat, 11 Mar 2017 00:07:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905906#comment-15905906

Manoj Govindassamy commented on HDFS-10530:

bq. isPlacementPolicySatisfied should use getRealTotalBlockNum

Patch 3 fix of changing the isPlacementPolicySatisfied() to make use getRealTotalBlockNum()
instead of getRealDataBlockNum() will definitely fix the blocks which are either detected
for reconstruction needed or the newly created files. But, for the ones which are already
created and mis-replicated, I am not seeing this patch recovering them. I will post a addendum
patch soon with the latest rebase and the modified test as explained in my previous comment.

> BlockManager reconstruction work scheduling should correctly adhere to EC block placement
> ------------------------------------------------------------------------------------------------
>                 Key: HDFS-10530
>                 URL: https://issues.apache.org/jira/browse/HDFS-10530
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Rui Gao
>            Assignee: Rui Gao
>              Labels: hdfs-ec-3.0-nice-to-have
>         Attachments: HDFS-10530.1.patch, HDFS-10530.2.patch, HDFS-10530.3.patch
> This issue was found by [~tfukudom].
> Under RS-DEFAULT-6-3-64k EC policy, 
> 1. Create an EC file, the file was witten to all the 5 racks( 2 dns for each) of the
> 2. Reconstruction work would be scheduled if the 6th rack is added. 
> 3. While adding the 7th rack or more racks will not trigger reconstruction work. 
> Based on default EC block placement policy defined in “BlockPlacementPolicyRackFaultTolerant.java”,
EC file should be able to be scheduled to distribute to 9 racks if possible.
> In *BlockManager#isPlacementPolicySatisfied(BlockInfo storedBlock)* , *numReplicas* of
striped blocks might should be *getRealTotalBlockNum()*, instead of *getRealDataBlockNum()*.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message