hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Karandikar <siddharth.karandi...@gmail.com>
Subject Re: Import HBase snapshots possible?
Date Thu, 01 Aug 2013 14:25:33 GMT
Its failing with '//' as well as '///'. Error suggests that it needs local fs.

3 ///
ssk01:~/siddharth/tools/hbase-0.95.1-hadoop1 # ./bin/hbase
org.apache.hadoop.hbase.snapshot.ExportSnapshot
-Dhbase.rootdir=hdfs:///10.209.17.88:9000/hbase/s2 -snapshot s2
-copy-to /root/siddharth/tools/hbase-0.95.1-hadoop1/data/
Exception in thread "main" java.io.IOException: Incomplete HDFS URI,
no host: hdfs:///10.209.17.88:9000/hbase/s2
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:85)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
        at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:860)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:594)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:690)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:694)

2 //

ssk01:~/siddharth/tools/hbase-0.95.1-hadoop1 # ./bin/hbase
org.apache.hadoop.hbase.snapshot.ExportSnapshot
-Dhbase.rootdir=hdfs://10.209.17.88:9000/hbase/s2 -snapshot s2
-copy-to /root/siddharth/tools/hbase-0.95.1-hadoop1/data/
Exception in thread "main" java.lang.IllegalArgumentException: Wrong
FS: hdfs://10.209.17.88:9000/hbase/s2/.hbase-snapshot/s2/.snapshotinfo,
expected: file:///
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
        at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
        at org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.readSnapshotInfo(SnapshotDescriptionUtils.java:296)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:371)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:618)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:690)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:694)




Btw, I tried one more thing. From my HDFS location, I just did a copy like -
ssk01:~/siddharth/tools/hadoop-1.1.2 # ./bin/hadoop fs -copyToLocal
hdfs://10.209.17.88:9000/hbase/s1/.hbase-snapshot/s1
/root/siddharth/tools/hbase-0.95.1-hadoop1/data/.hbase-snapshot/

After doing this, I am able to see s1 in 'list_snapshots'. But it is
failing at 'clone_snapshot'.

hbase(main):014:0> clone_snapshot 's1', 'ts1'

ERROR: java.io.IOException: Table 'ts1' not yet enabled, after 199617ms.

Here is some help for this command:
Create a new table by cloning the snapshot content.
There're no copies of data involved.
And writing on the newly created table will not influence the snapshot data.

Examples:
  hbase> clone_snapshot 'snapshotName', 'tableName'



On Thu, Aug 1, 2013 at 7:44 PM, Matteo Bertozzi <theo.bertozzi@gmail.com> wrote:
> you have to use 3 slashes otherwise is interpreted as local file-system path
> -Dhbase.rootdir=hdfs:///10.209.17.88:9000/hbase
>
> Matteo
>
>
>
> On Thu, Aug 1, 2013 at 3:09 PM, Siddharth Karandikar <
> siddharth.karandikar@gmail.com> wrote:
>
>> Tried what you suggested. Here is what I get -
>>
>> ssk01:~/siddharth/tools/hbase-0.95.1-hadoop1 # ./bin/hbase
>> org.apache.hadoop.hbase.snapshot.ExportSnapshot
>> -Dhbase.rootdir=hdfs://10.209.17.88:9000/hbase -snapshot s1 -copy-to
>> /root/siddharth/tools/hbase-0.95.1-hadoop1/data/
>> Exception in thread "main" java.lang.IllegalArgumentException: Wrong
>> FS: hdfs://10.209.17.88:9000/hbase/.hbase-snapshot/s1/.snapshotinfo,
>> expected: file:///
>>         at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
>>         at
>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
>>         at
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
>>         at
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>>         at
>> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
>>         at
>> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
>>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
>>         at
>> org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.readSnapshotInfo(SnapshotDescriptionUtils.java:296)
>>         at
>> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:371)
>>         at
>> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:618)
>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>         at
>> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:690)
>>         at
>> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:694)
>>
>>
>> Am I missing something?
>>
>>
>> Thanks,
>> Siddharth
>>
>>
>> On Thu, Aug 1, 2013 at 7:31 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>
>> wrote:
>> > Ok, so to export a snapshot from your HBase cluster, you can do
>> > $ bin/hbase class org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot
>> > -snapshot MySnapshot -copy-to hdfs:///srv2:8082/my-backup-dir
>> >
>> > Now on cluster2, hdfs:///srv2:8082 you've your my-backup-dir that
>> contains
>> > the exported snapshot (note that the snapshot is under hidden dirs
>> > .snapshots, and .archive)
>> >
>> > Now if you want to restore the snapshot,  you have to export it back an
>> > HBase cluster.
>> > So on cluster2, you can do:
>> > $ bin/hbase class org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot -D
>> > hbase.rootdir=hdfs:///srv2:8082/my-backup-dir -snapshot MySnapshot
>> -copy-to
>> > hdfs:///hbaseSrv:8082/hbase
>> >
>> >
>> > so, to recap
>> >  - You take a snapshot
>> >  - You Export the snapshot from HBase Cluster-1 -> to a simple HDFS dir
>> in
>> > Cluster-2
>> >  - Then you want to restore
>> >  - You Export the snapshot from HDFS dir in Cluster-2 to HBase Cluster
>> (it
>> > can be a different one from the original)
>> >  - From the hbase shell you can just: clone_snapshot 'snapshotName',
>> > 'newTableName' if the table does not exists or use restore_snapshot
>> > 'snapshotName', if there's a table with the same name
>> >
>> >
>> > Matteo
>> >
>> >
>> >
>> > On Thu, Aug 1, 2013 at 2:54 PM, Siddharth Karandikar <
>> > siddharth.karandikar@gmail.com> wrote:
>> >
>> >> Yeah, thats right. But the issue is, hdfs that I am exporting to is
>> >> not under HBase.
>> >> Can you please provide some example command to do this...
>> >>
>> >>
>> >> Thanks,
>> >> Siddharth
>> >>
>> >> On Thu, Aug 1, 2013 at 7:17 PM, Matteo Bertozzi <
>> theo.bertozzi@gmail.com>
>> >> wrote:
>> >> > Yes, the export an HDFS path.
>> >> > $ bin/hbase class org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot
>> >> > -snapshot MySnapshot -copy-to hdfs:///srv2:8082/hbase
>> >> >
>> >> > so you can export to some /my-backup-dir on your HDFS
>> >> > and then you've to export back to an hbase cluster, when you want to
>> >> > restore it
>> >> >
>> >> > Matteo
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Aug 1, 2013 at 2:45 PM, Siddharth Karandikar <
>> >> > siddharth.karandikar@gmail.com> wrote:
>> >> >
>> >> >> Can't I export it to plain HDFS? I think that would be very useful.
>> >> >>
>> >> >> On Thu, Aug 1, 2013 at 7:08 PM, Matteo Bertozzi <
>> >> theo.bertozzi@gmail.com>
>> >> >> wrote:
>> >> >> > The ExportSnapshot will export the snapshot data+metadata,
in
>> theory,
>> >> to
>> >> >> > another hbase cluster.
>> >> >> > so on the second cluster you'll now be able to do "list_snapshots"
>> >> from
>> >> >> > shell and see the exported snapshot.
>> >> >> > now you can simply do clone_snapshot "snapshot_name",
>> "new_table_name"
>> >> >> and
>> >> >> > you're restoring a snapshot on the second cluster
>> >> >> >
>> >> >> > assuming that you have removed the snapshot from cluster1
and you
>> >> want to
>> >> >> > export back your snapshot..
>> >> >> > you just use ExportSnapshot again to move the snapshot from
>> cluster2
>> >> to
>> >> >> > cluster1
>> >> >> > and same as before you do a clone_snapshot to restore it
>> >> >> >
>> >> >> > Matteo
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Aug 1, 2013 at 2:35 PM, Siddharth Karandikar <
>> >> >> > siddharth.karandikar@gmail.com> wrote:
>> >> >> >
>> >> >> >> Hi there,
>> >> >> >>
>> >> >> >> I am testing out newly added snapshot capability, ExportSnapshot
>> in
>> >> >> >> particular.
>> >> >> >> Its working fine for me. I am able to run ExportSnapshot
properly.
>> >> >> >>
>> >> >> >> But the biggest (noob) issue is, once exported, is there
any way
>> to
>> >> >> >> import those snapshots back in hbase? I don't see any
>> ImportSnapshot
>> >> >> >> util there.
>> >> >> >>
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> Siddharth
>> >> >> >>
>> >> >>
>> >>
>>

Mime
View raw message