hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5907) Handle block deletion requests during rolling upgrades
Date Wed, 12 Feb 2014 06:55:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898871#comment-13898871

Arpit Agarwal commented on HDFS-5907:

Hi Suresh, I described it in earlier comment. The approach is the same.

The patch adds a trash directory under BlockPoolSliceStorage. The trash directory location
is SD/current/BP-.../trash/ and its sub-directory layout mirrors that of SD/current/BP-.../current/.
This allows keeping memory of where to restore the file on rollback.
An alternative is to just rename the file under its current location to indicate it is a deleted
block but this makes finalize and rollback expensive as we may have to potentially iterate
over millions of block files.
The DN detects whether a rolling upgrade is in progress or not by looking for the presence
of RollingUpgradeStatus in HeartbeatResponse.
If a block is deleted for a BPID undergoing rolling upgrade then it is moved to a trash directory
instead. On finalize the trash directory is emptied. On rollback, files under trash/ are restored
to their corresponding locations under current/.
The only addition is that DN rollback is triggered by passing a {{-rollingUpgrade rollback}}
flag at startup. This was chosen to be the same as the flag passed to NN for rollback.

> Handle block deletion requests during rolling upgrades
> ------------------------------------------------------
>                 Key: HDFS-5907
>                 URL: https://issues.apache.org/jira/browse/HDFS-5907
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: HDFS-5535 (Rolling upgrades)
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-5907.01.patch, HDFS-5907.02.patch
> DN changes when a rolling upgrade is in progress:
> # DataNode should handle block deletions by moving block files to 'trash'.
> # Block files should be restored to their original locations during a rollback.
> # Purge trash when the rolling upgrade is finalized.

This message was sent by Atlassian JIRA

View raw message