hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11248) [SPS]: Handle partial block location movements
Date Thu, 29 Dec 2016 07:21:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784738#comment-15784738

Uma Maheswara Rao G commented on HDFS-11248:

+1 on the latest patch

> [SPS]: Handle partial block location movements
> ----------------------------------------------
>                 Key: HDFS-11248
>                 URL: https://issues.apache.org/jira/browse/HDFS-11248
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>    Affects Versions: HDFS-10285
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>         Attachments: HDFS-11248-HDFS-10285-00.patch, HDFS-11248-HDFS-10285-01.patch,
HDFS-11248-HDFS-10285-02.patch, HDFS-11248-HDFS-10285-03.patch, HDFS-11248-HDFS-10285-04.patch,
> This jira is to handle partial block location movements due to unavailability of target
nodes for the matching storage type. 
> For example, We have only A(disk,archive), B(disk) and C(disk,archive) are live nodes
with A & C have archive storage type. Say, we have a block with locations {{A(disk), B(disk),
C(disk)}}. Again assume, user changed the storage policy to COLD. Now, SPS internally starts
preparing the src-target pairing like, {{src=> (A, B, C) and target=> (A, C)}} and sends
BLOCK_STORAGE_MOVEMENT to the coordinator. SPS is skipping B as it doesn't have archive media
to indicate that it should do retries to satisfy all block locations after some time. On receiving
the movement command, coordinator will pair the src-target node to schedule actual physical
movements like, {{movetask=> (A, A), (B, C)}}. Here ideally it should do {{(C, C)}} instead
of {{(B, C)}} but mistakenly choosing the source C and creates problem.
> IMHO, the implicit assumptions of retry needed is creating confusions and leads to coding
mistakes. One idea to fix this problem is to create a new flag {{retryNeeded}} flag to make
it more readable. With this, SPS will prepare only the matching pair and dummy source slots
will be avoided like, {{src=> (A, C) and target=> (A, C)}} and mark {{retryNeeded=true}}
to convey the message that this {{trackId}} has only partial blocks movements.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message