hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matteo Bertozzi <theo.berto...@gmail.com>
Subject Re: Import HBase snapshots possible?
Date Thu, 01 Aug 2013 14:44:10 GMT
You can't just copy the .snapshot folder... so now you've the RSs that is
failing since the files for the cloned table are not available..

when you specify the hbase.rootdir you've just to specify the hbase.rootdir
the one in /etc/hbase-site.xml which doesn't contain the name of the
snapshot/table that you want to export (e.g.
hdfs://10.209.17.88:9000/hbasenot hdfs://
10.209.17.88:9000/hbase/s2)



Matteo



On Thu, Aug 1, 2013 at 3:25 PM, Siddharth Karandikar <
siddharth.karandikar@gmail.com> wrote:

> Its failing with '//' as well as '///'. Error suggests that it needs local
> fs.
>
> 3 ///
> ssk01:~/siddharth/tools/hbase-0.95.1-hadoop1 # ./bin/hbase
> org.apache.hadoop.hbase.snapshot.ExportSnapshot
> -Dhbase.rootdir=hdfs:///10.209.17.88:9000/hbase/s2 -snapshot s2
> -copy-to /root/siddharth/tools/hbase-0.95.1-hadoop1/data/
> Exception in thread "main" java.io.IOException: Incomplete HDFS URI,
> no host: hdfs:///10.209.17.88:9000/hbase/s2
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:85)
>         at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
>         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
>         at
> org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:860)
>         at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:594)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:690)
>         at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:694)
>
> 2 //
>
> ssk01:~/siddharth/tools/hbase-0.95.1-hadoop1 # ./bin/hbase
> org.apache.hadoop.hbase.snapshot.ExportSnapshot
> -Dhbase.rootdir=hdfs://10.209.17.88:9000/hbase/s2 -snapshot s2
> -copy-to /root/siddharth/tools/hbase-0.95.1-hadoop1/data/
> Exception in thread "main" java.lang.IllegalArgumentException: Wrong
> FS: hdfs://10.209.17.88:9000/hbase/s2/.hbase-snapshot/s2/.snapshotinfo,
> expected: file:///
>         at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
>         at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
>         at
> org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.readSnapshotInfo(SnapshotDescriptionUtils.java:296)
>         at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:371)
>         at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:618)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:690)
>         at
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:694)
>
>
>
>
> Btw, I tried one more thing. From my HDFS location, I just did a copy like
> -
> ssk01:~/siddharth/tools/hadoop-1.1.2 # ./bin/hadoop fs -copyToLocal
> hdfs://10.209.17.88:9000/hbase/s1/.hbase-snapshot/s1
> /root/siddharth/tools/hbase-0.95.1-hadoop1/data/.hbase-snapshot/
>
> After doing this, I am able to see s1 in 'list_snapshots'. But it is
> failing at 'clone_snapshot'.
>
> hbase(main):014:0> clone_snapshot 's1', 'ts1'
>
> ERROR: java.io.IOException: Table 'ts1' not yet enabled, after 199617ms.
>
> Here is some help for this command:
> Create a new table by cloning the snapshot content.
> There're no copies of data involved.
> And writing on the newly created table will not influence the snapshot
> data.
>
> Examples:
>   hbase> clone_snapshot 'snapshotName', 'tableName'
>
>
>
> On Thu, Aug 1, 2013 at 7:44 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>
> wrote:
> > you have to use 3 slashes otherwise is interpreted as local file-system
> path
> > -Dhbase.rootdir=hdfs:///10.209.17.88:9000/hbase
> >
> > Matteo
> >
> >
> >
> > On Thu, Aug 1, 2013 at 3:09 PM, Siddharth Karandikar <
> > siddharth.karandikar@gmail.com> wrote:
> >
> >> Tried what you suggested. Here is what I get -
> >>
> >> ssk01:~/siddharth/tools/hbase-0.95.1-hadoop1 # ./bin/hbase
> >> org.apache.hadoop.hbase.snapshot.ExportSnapshot
> >> -Dhbase.rootdir=hdfs://10.209.17.88:9000/hbase -snapshot s1 -copy-to
> >> /root/siddharth/tools/hbase-0.95.1-hadoop1/data/
> >> Exception in thread "main" java.lang.IllegalArgumentException: Wrong
> >> FS: hdfs://10.209.17.88:9000/hbase/.hbase-snapshot/s1/.snapshotinfo,
> >> expected: file:///
> >>         at
> org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
> >>         at
> >>
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
> >>         at
> >>
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
> >>         at
> >>
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> >>         at
> >>
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
> >>         at
> >>
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
> >>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
> >>         at
> >>
> org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.readSnapshotInfo(SnapshotDescriptionUtils.java:296)
> >>         at
> >>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:371)
> >>         at
> >>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:618)
> >>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >>         at
> >>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:690)
> >>         at
> >>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:694)
> >>
> >>
> >> Am I missing something?
> >>
> >>
> >> Thanks,
> >> Siddharth
> >>
> >>
> >> On Thu, Aug 1, 2013 at 7:31 PM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>
> >> wrote:
> >> > Ok, so to export a snapshot from your HBase cluster, you can do
> >> > $ bin/hbase class org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot
> >> > -snapshot MySnapshot -copy-to hdfs:///srv2:8082/my-backup-dir
> >> >
> >> > Now on cluster2, hdfs:///srv2:8082 you've your my-backup-dir that
> >> contains
> >> > the exported snapshot (note that the snapshot is under hidden dirs
> >> > .snapshots, and .archive)
> >> >
> >> > Now if you want to restore the snapshot,  you have to export it back
> an
> >> > HBase cluster.
> >> > So on cluster2, you can do:
> >> > $ bin/hbase class
> org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot -D
> >> > hbase.rootdir=hdfs:///srv2:8082/my-backup-dir -snapshot MySnapshot
> >> -copy-to
> >> > hdfs:///hbaseSrv:8082/hbase
> >> >
> >> >
> >> > so, to recap
> >> >  - You take a snapshot
> >> >  - You Export the snapshot from HBase Cluster-1 -> to a simple HDFS
> dir
> >> in
> >> > Cluster-2
> >> >  - Then you want to restore
> >> >  - You Export the snapshot from HDFS dir in Cluster-2 to HBase Cluster
> >> (it
> >> > can be a different one from the original)
> >> >  - From the hbase shell you can just: clone_snapshot 'snapshotName',
> >> > 'newTableName' if the table does not exists or use restore_snapshot
> >> > 'snapshotName', if there's a table with the same name
> >> >
> >> >
> >> > Matteo
> >> >
> >> >
> >> >
> >> > On Thu, Aug 1, 2013 at 2:54 PM, Siddharth Karandikar <
> >> > siddharth.karandikar@gmail.com> wrote:
> >> >
> >> >> Yeah, thats right. But the issue is, hdfs that I am exporting to is
> >> >> not under HBase.
> >> >> Can you please provide some example command to do this...
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Siddharth
> >> >>
> >> >> On Thu, Aug 1, 2013 at 7:17 PM, Matteo Bertozzi <
> >> theo.bertozzi@gmail.com>
> >> >> wrote:
> >> >> > Yes, the export an HDFS path.
> >> >> > $ bin/hbase class
> org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot
> >> >> > -snapshot MySnapshot -copy-to hdfs:///srv2:8082/hbase
> >> >> >
> >> >> > so you can export to some /my-backup-dir on your HDFS
> >> >> > and then you've to export back to an hbase cluster, when you want
> to
> >> >> > restore it
> >> >> >
> >> >> > Matteo
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Thu, Aug 1, 2013 at 2:45 PM, Siddharth Karandikar <
> >> >> > siddharth.karandikar@gmail.com> wrote:
> >> >> >
> >> >> >> Can't I export it to plain HDFS? I think that would be very
> useful.
> >> >> >>
> >> >> >> On Thu, Aug 1, 2013 at 7:08 PM, Matteo Bertozzi <
> >> >> theo.bertozzi@gmail.com>
> >> >> >> wrote:
> >> >> >> > The ExportSnapshot will export the snapshot data+metadata,
in
> >> theory,
> >> >> to
> >> >> >> > another hbase cluster.
> >> >> >> > so on the second cluster you'll now be able to do
> "list_snapshots"
> >> >> from
> >> >> >> > shell and see the exported snapshot.
> >> >> >> > now you can simply do clone_snapshot "snapshot_name",
> >> "new_table_name"
> >> >> >> and
> >> >> >> > you're restoring a snapshot on the second cluster
> >> >> >> >
> >> >> >> > assuming that you have removed the snapshot from cluster1
and
> you
> >> >> want to
> >> >> >> > export back your snapshot..
> >> >> >> > you just use ExportSnapshot again to move the snapshot
from
> >> cluster2
> >> >> to
> >> >> >> > cluster1
> >> >> >> > and same as before you do a clone_snapshot to restore
it
> >> >> >> >
> >> >> >> > Matteo
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Thu, Aug 1, 2013 at 2:35 PM, Siddharth Karandikar
<
> >> >> >> > siddharth.karandikar@gmail.com> wrote:
> >> >> >> >
> >> >> >> >> Hi there,
> >> >> >> >>
> >> >> >> >> I am testing out newly added snapshot capability,
> ExportSnapshot
> >> in
> >> >> >> >> particular.
> >> >> >> >> Its working fine for me. I am able to run ExportSnapshot
> properly.
> >> >> >> >>
> >> >> >> >> But the biggest (noob) issue is, once exported, is
there any
> way
> >> to
> >> >> >> >> import those snapshots back in hbase? I don't see
any
> >> ImportSnapshot
> >> >> >> >> util there.
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> Siddharth
> >> >> >> >>
> >> >> >>
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message