hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7621) Erasure Coding: update the Balancer/Mover data migration logic
Date Wed, 22 Apr 2015 03:54:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506368#comment-14506368

Walter Su commented on HDFS-7621:

001 patch is initial patch. No tests included. I want to know if the logic works.

What does the patch do:
1. {{Balancer.isGoodBlockCandidate(..)}} is a guard function. The patch make the function
take EC block group into consideration.  It can avoid place 2 block ( of BlockGroup ) in same
node. {{reduceNumOfRacks(..)}} guard can avoid reduce number of racks after movement, so it
tries not to violate (specific) placement policy but can't guarantee.

How does the patch work:
Balancer doesn't know EC Block group, it only know reported block( with real blockId in datanode).
But I want it to take the locations of the group into consideration when choosing targets.
It requires the correctness of {{DBlock.isLocatedOn(..)}} HDFS-8147 .

> Erasure Coding: update the Balancer/Mover data migration logic
> --------------------------------------------------------------
>                 Key: HDFS-7621
>                 URL: https://issues.apache.org/jira/browse/HDFS-7621
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Walter Su
>         Attachments: HDFS-7621.001.patch
> Currently the Balancer/Mover only considers the distribution of replicas of the same
block during data migration: the migration cannot decrease the number of racks. With EC the
Balancer and Mover should also take into account the distribution of blocks belonging to the
same block group.

This message was sent by Atlassian JIRA

View raw message