hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6233) [brainstorm] snapshots: hardlink alternatives
Date Sat, 07 Jul 2012 20:00:48 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408762#comment-13408762

stack commented on HBASE-6233:

@Matteo Thanks for taking the time to do the writeup.  Helpful.  I like how your symlink work
would make it so no work moving up on to hdfs hard links.

I was wondering if you have any concern around creation of all the symlinks on a table of
some decent size taking a good bit of time Matteo?  The window during which the snapshot is
being made could be pretty wide.  Would that be a problem?

You ask for ideas and the only one I have is the hackneyed one copied from bdbje where on
compaction, we do not delete files; rather we just rename them w/ a '.del' ending and leave
them in place.  On snapshot, we make a manifest of all files in the table.  On restore, we'd
read the manifest and look for files first w/o the .del and then if not found, with the .del.
 I've not thought it all through to the extent of your attached pdf -- I can see how it could
get tangled pretty quickly -- but throwing it up there since you were asking.

bq. ...and can be an external tool or an internal thread that scan the snapshot.

Could hitch a ride on the current meta scanner, the one that cleans the parent regions from

Adding list of files to .META. might make for our being able to do other fancyness such as
the Accumulo fast table copy, etc.

Let me read your doc. some more (and Jesse's work).

> [brainstorm] snapshots: hardlink alternatives
> ---------------------------------------------
>                 Key: HBASE-6233
>                 URL: https://issues.apache.org/jira/browse/HBASE-6233
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: Matteo Bertozzi
>            Assignee: Matteo Bertozzi
>         Attachments: Restore-Snapshot-Hardlink-alternatives.pdf
> Discussion ticket around snapshots and hardlink alternatives.
> (See the HDFS-3370 discussion about hardlink and implementation problems)
> (taking for a moment WAL out of the discussion and focusing on hfiles)
> With hardlinks available taking snapshot will be fairly easy:
> * (hfiles are immutable)
> * hardlink to .snapshot/name to take snapshot
> * hardlink from .snapshot/name to restore the snapshot
> * No code change needed (on fs.delete() only one reference is deleted)
> but we don't have hardlinks, what are the alternatives?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message