hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hanisha Koneru (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12996) DataNode Replica Trash
Date Wed, 10 Jan 2018 23:29:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321337#comment-16321337

Hanisha Koneru commented on HDFS-12996:

Thanks for the review, [~shahrs87].

bq. Suppose user1 and user2 deleted some of their directories (lets say dir1 and dir2 respectively).
If user1 wants to recover its directory, then we will recover dir2 as well ?
Yes. In the current design, recovery is done by rolling back to an earlier image. We could
separately build a more fine-grained recovery mechanism on top of the replica trash.
bq. Many of our clients(lets say user1) use /tmp/<userId> to store their intermediate
task output (to work around quota problems). After a job completes, they delete this space
and use the same location to store next job output. In the meantime if some other user(lets
say user2) wants to recover their mistakenly deleted directory then we will go back in time
for user1 which might corrupt user1's output directory.
True. This again would be a trade-off between recovering the deleted data and undoing operations
performed after the delete operation. Only an administrator can make this call.
The goal of this feature is to provide a safe-guard to recover from catastrophic mistakes
where it is acceptable to lose a few recent changes to recover deleted data.
bq. Also the design looks very similar to Checkpointing/Snapshots.
--> Didn't get what you mean by checkpointing in this context. If you take frequent rolling
snapshots e.g. hourly snapshots of root directory from a cron job, then you don't need this
feature and you can recover deleted files from a recent snapshot. However very few clusters
are setup for this.

> DataNode Replica Trash
> ----------------------
>                 Key: HDFS-12996
>                 URL: https://issues.apache.org/jira/browse/HDFS-12996
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Hanisha Koneru
>            Assignee: Hanisha Koneru
>         Attachments: DataNode_Replica_Trash_Design_Doc.pdf
> DataNode Replica Trash will allow administrators to recover from a recent delete request
that resulted in catastrophic loss of user data. This is achieved by placing all invalidated
blocks in a replica trash on the datanode before completely purging them from the system.
The design doc is attached here.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message