hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5907) Handle block deletion requests during rolling upgrades
Date Tue, 11 Feb 2014 23:08:20 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arpit Agarwal updated HDFS-5907:

    Attachment: HDFS-5907.01.patch

Preliminary patch, needs test cases.

I was able to test delete/finalize manually. Was not able to test rollback yet.

The patch adds a trash directory under {{BlockPoolSliceStorage}}. The trash directory location
is {{SD/current/BP-.../trash/}} and its sub-directory layout mirrors that of {{SD/current/BP-.../current/}}.
This allows keeping memory of where to restore the file on rollback.

An alternative is to just rename the file under its current location to indicate it is a deleted
block but this makes rollback expensive as we may have to potentially iterate over millions
of block files.

The DN detects whether a rolling upgrade is in progress or not by looking for the presence
of {{RollingUpgradeStatus}} in {{HeartbeatResponse}}.
If a block is deleted for a BPID undergoing rolling upgrade then it is moved to a trash directory
instead. On finalize the trash directory is emptied. On rollback, files under trash/ are restored
to their corresponding locations under current/.

> Handle block deletion requests during rolling upgrades
> ------------------------------------------------------
>                 Key: HDFS-5907
>                 URL: https://issues.apache.org/jira/browse/HDFS-5907
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: HDFS-5535 (Rolling upgrades)
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-5907.01.patch
> Jira to track the following:
> # During a rolling upgrade the DataNode should handle block deletions by creating hard
links to the block files.
> # Files must be restored during a rollback.
> # Cleanup hard links during finalize.

This message was sent by Atlassian JIRA

View raw message