hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7621) Erasure Coding: update the Balancer/Mover data migration logic
Date Thu, 28 May 2015 06:27:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562357#comment-14562357

Zhe Zhang commented on HDFS-7621:

Sorry for coming to this review late.

Again, the patch looks reasonable overall. My biggest concern is that the patch changes {{Dispatcher}}
pretty heavily in place. It'd be better if we leave the current logic mostly unchanged and
add striping logic incrementally. This will make the merge review much cleaner and easier.

{{DBlock}} actually just maps a {{Block}} to locations. The added {{DBlockStriped}} does 2
things: 1) maps {{Block}} to locations (inherited from {{DBlock}}); and 2) when given a {{StorageGroup}},
calculates the index of the internal block. How about creating a structure only for purpose
#2? Something like the below:

private static class GlobalBlockGroupMap {
  private final Map<Block, StripedBlockWithLocations> map = new HashMap<>();
  private int getInternalBlkIdx(Block b, StorageGroup storage) {

Then the change to existing {{Dispatcher}} code can be simple. Right before sending a move
command to DN, just check the new map to see if we need to translate the block ID. It seems
we can add the check in {{executePendingMove}} for this purpose, but I'm not 100% sure.

> Erasure Coding: update the Balancer/Mover data migration logic
> --------------------------------------------------------------
>                 Key: HDFS-7621
>                 URL: https://issues.apache.org/jira/browse/HDFS-7621
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Walter Su
>              Labels: HDFS-7285
>         Attachments: HDFS-7621.001.patch, HDFS-7621.002.patch, HDFS-7621.003.patch, HDFS-7621.004.patch,
HDFS-7621.005.patch, HDFS-7621.006.patch
> Currently the Balancer/Mover only considers the distribution of replicas of the same
block during data migration: the migration cannot decrease the number of racks. With EC the
Balancer and Mover should also take into account the distribution of blocks belonging to the
same block group.

This message was sent by Atlassian JIRA

View raw message