hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11248) [SPS]: Handle partial block location movements
Date Thu, 29 Dec 2016 02:19:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784281#comment-15784281
] 

Rakesh R commented on HDFS-11248:
---------------------------------

Thanks [~umamaheswararao] for the detailed reviews. IIUC, the following log message in my
previous patch has to be corrected. Attached another patch addressing the same.
{code}
LOG.info("Blocks storage movement results for the tracking id : "
                + storageMovementAttemptedResult.getTrackId()
                + " is reported from co-ordinating datanode. "
                + "The result status is SUCCESS.”);
{code}

> [SPS]: Handle partial block location movements
> ----------------------------------------------
>
>                 Key: HDFS-11248
>                 URL: https://issues.apache.org/jira/browse/HDFS-11248
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>    Affects Versions: HDFS-10285
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>         Attachments: HDFS-11248-HDFS-10285-00.patch, HDFS-11248-HDFS-10285-01.patch,
HDFS-11248-HDFS-10285-02.patch, HDFS-11248-HDFS-10285-03.patch, HDFS-11248-HDFS-10285-04.patch,
HDFS-11248-HDFS-10285-05.patch
>
>
> This jira is to handle partial block location movements due to unavailability of target
nodes for the matching storage type. 
> For example, We have only A(disk,archive), B(disk) and C(disk,archive) are live nodes
with A & C have archive storage type. Say, we have a block with locations {{A(disk), B(disk),
C(disk)}}. Again assume, user changed the storage policy to COLD. Now, SPS internally starts
preparing the src-target pairing like, {{src=> (A, B, C) and target=> (A, C)}} and sends
BLOCK_STORAGE_MOVEMENT to the coordinator. SPS is skipping B as it doesn't have archive media
to indicate that it should do retries to satisfy all block locations after some time. On receiving
the movement command, coordinator will pair the src-target node to schedule actual physical
movements like, {{movetask=> (A, A), (B, C)}}. Here ideally it should do {{(C, C)}} instead
of {{(B, C)}} but mistakenly choosing the source C and creates problem.
> IMHO, the implicit assumptions of retry needed is creating confusions and leads to coding
mistakes. One idea to fix this problem is to create a new flag {{retryNeeded}} flag to make
it more readable. With this, SPS will prepare only the matching pair and dummy source slots
will be avoided like, {{src=> (A, C) and target=> (A, C)}} and mark {{retryNeeded=true}}
to convey the message that this {{trackId}} has only partial blocks movements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message