hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2802) Support for RW/RO snapshots in HDFS
Date Wed, 31 Oct 2012 01:55:12 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Suresh Srinivas updated HDFS-2802:

    Attachment: Snapshots20121030.pdf

Attaching the updated design document. Hopefully it addresses the issues that have been rasied.
This should give sufficient details about the implementation we are currently working on.
To summarize:

# *Snapshot allowed only at the root vs snapshot at the subdirectories* - Ability to snapshot
a sub-directory is very important requirement for many Hadoop users. Please see the requirements
in the document posted for more details. The alternate proposal to allow snapshots only at
the root is a non-starter in this regard.
# *Efficiency of snapshot creation and management* - The current design addresses the concerns
raised. To summarize the creation of snapshot is O(1). The design uses copy-on-modify approach
so that the cost of snapshot is zero when there is no modification and is proportional the
modifications when they are made. Please provide feedback.
# *Snapshot of being written files and consistency*  - Our document describes several design
choices - some easy, some complicated. Please see the proposed choice in the document. We
could continue this discussion in HDFS-3960.

I took a look at the alternate proposal. It is too high level without sufficient details to
evaluate. From my limited understanding of the alternate proposal, the design document we
have posted here has several significant advantages over it:
# It supports sub-directory snapshots, an important use case for many Hadoop users.
# It supports on-demand and user managed snapshots.
# When snapshots are not created, there is no cost incurred in terms of memory. The alternate
proposal has O(N) memory cost for storing tags.
# Our design can also be extended to do RW snapshots, if we feel a need for it.

Hopefully this clarifies the design better. We would like continue focus on implementing it.
Any feedback provided will be incorporated into the design and the implementation.

> Support for RW/RO snapshots in HDFS
> -----------------------------------
>                 Key: HDFS-2802
>                 URL: https://issues.apache.org/jira/browse/HDFS-2802
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, name-node
>            Reporter: Hari Mankude
>            Assignee: Hari Mankude
>         Attachments: HDFSSnapshotsDesign.pdf, snap.patch, snapshot-one-pager.pdf, Snapshots20121018.pdf,
> Snapshots are point in time images of parts of the filesystem or the entire filesystem.
Snapshots can be a read-only or a read-write point in time copy of the filesystem. There are
several use cases for snapshots in HDFS. I will post a detailed write-up soon with with more

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message