hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-50) Snapshot of table
Date Thu, 01 Apr 2010 14:08:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852393#action_12852393

stack commented on HBASE-50:

Let me freshen this issue by synopsizing what has gone before and adding some new remarks
(This issue has been labeled gsoc so candidates are taking a look trying to figure whats involved).

The current state of a table is comprised of the memstore content of all currently running
regionservers and the content of the /${hbase.rootdir}/TABLENAME directory in HDFS.  There
is also metadata -- mainly the current state of its schema and the regions of which it is
comprised -- kept up in .META. "catalog" table.

In the above, there is talk of  'offlining' or flipping the table 'read-only'.  Realistically
we cannot require a production hbase be offlined or even flipped read-only making a snapshot.

So, snapshotting needs to something like Clint Morgan describes above where we don't depend
on memstore flush, where we instead get the memstore content from WAL files, and where snapshotting
is triggered by sending a signal across the cluster, probably best distributed via zookeeper.
 On receipt, regionservers roll their WAL and dump a manifest of all the files they have in
their care; e.g. a list of all regions they are currently serving and the files of which these
are comprised.  The regionserver hosting the .META. would also dump the metadata for said
table at this time.  During this WAL roll and manifest dump, we would continue to take on
writes but there might be some transitions we'd suspend for the short time it would take to
write out manifest, etc., such as memstore flushing.

So a snapshot can be saved aside or reconstituted in-situ, regionservers cannot delete files.
 For example, a compaction instead moves the old files aside rather than delete them when
the new compacted file is made.  The moved aside, non-deleted files might be copied elsewhere
by some other process (a distcp MR job).  They might also be cleaned up by a cleaner that
could distingush old snapshots from the more recent (be careful not to remove files still
in use).

We'd need a script, something like add_table.rb that could fixup .META. with the snapshotted
.META. state and that moved files around in the filesystem -- from snapshot location back
into the running position.

> Snapshot of table
> -----------------
>                 Key: HBASE-50
>                 URL: https://issues.apache.org/jira/browse/HBASE-50
>             Project: Hadoop HBase
>          Issue Type: New Feature
>            Reporter: Billy Pearson
>            Priority: Minor
> Havening an option to take a snapshot of a table would be vary useful in production.
> What I would like to see this option do is do a merge of all the data into one or more
files stored in the same folder on the dfs. This way we could save data in case of a software
bug in hadoop or user code. 
> The other advantage would be to be able to export a table to multi locations. Say I had
a read_only table that must be online. I could take a snapshot of it when needed and export
it to a separate data center and have it loaded there and then i would have it online at multi
data centers for load balancing and failover.
> I understand that hadoop takes the need out of havening backup to protect from failed
servers, but this does not protect use from software bugs that might delete or alter data
in ways we did not plan. We should have a way we can roll back a dataset.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message