hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5907) Handle block deletion requests during rolling upgrades
Date Wed, 12 Feb 2014 23:02:20 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arpit Agarwal updated HDFS-5907:

    Attachment: HDFS-5907.04.patch

Thanks for the detailed review [~sureshms]. Addressed everything not mentioned below and also
feedback from [~vinayrpet].

Why is on finalize the blocks in trash are not being deleted?
They are deleted when NN signals finalize by the absence of {{RollingUpgradeStatus}} in the
{{HeartbeatResponse}}. This triggers purge via {{BPOfferService#signalRollingUpgrade}}.

Sorry I missed the logic of ENUM_WITH_ROLLING_UPGRAE_OPTION and its been a while, but I also
forgot about why we added DFS_DATANODE_STARTUP_KEY in the first place.
The logic for {{ENUM_WITH_ROLLING_UPGRADE_OPTION}} is required because the {{enum.toString}}
was changed separately to include {{RollingUpgradeStartupOption}} and we need to have a corresponding
parse method. This is the static {{#getEnum}}.

Also the logic is not quite right. restoreDirectory is always going to be null right. Checking
for null seems redundant and also you can just rename the trash to getRestoreDirectory(child).
Good catch, the assignment to null should have been outside the for loop, this is to avoid
recreating it if multiple times when restoring more than one child in the same parent. Fixed

Unit tests required for patterns added to BlockPoolSliceStorage, and methods getTrashDirectory,
getRestoreDirectory. Also these methods might be easier to test, if it accepts string
I've added a number of unit tests in this version of the patch. Still working on an E2E test
for DN finalize/rollback.

> Handle block deletion requests during rolling upgrades
> ------------------------------------------------------
>                 Key: HDFS-5907
>                 URL: https://issues.apache.org/jira/browse/HDFS-5907
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: HDFS-5535 (Rolling upgrades)
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-5907.01.patch, HDFS-5907.02.patch, HDFS-5907.04.patch
> DN changes when a rolling upgrade is in progress:
> # DataNode should handle block deletions by moving block files to 'trash'.
> # Block files should be restored to their original locations during a rollback.
> # Purge trash when the rolling upgrade is finalized.

This message was sent by Atlassian JIRA

View raw message