hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Export HBase snapshot to S3 creates empty root directory (prefix)
Date Thu, 05 May 2016 04:07:42 GMT
Lex:
Please also see this thread about s3n versus s3a:

http://search-hadoop.com/m/uOzYtE1Fy22eEWfe1&subj=Re+S3+Hadoop+FileSystems

On Wed, May 4, 2016 at 9:01 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>
wrote:

> never seen that problem before, but a couple of suggestions you can try.
>
> Instead of the old s3 driver, you can use s3n or s3a if you have it
> available (those are the ones I tested)
> and instead of using hbase.root dir use -copy-from
>
> ExportSnapshot -snapshot SNAPSHOT_NAME -copy-to s3a://BUCKET/NAMESPACE
> ExportSnapshot -snapshot SNAPSHOT_NAME -copy-from s3a://BUCKET/NAMESPACE
> -copy-to hdfs://HOST/hbase
>
> you can take a look at some in-progress doc about s3 and snapshot here:
> https://issues.apache.org/jira/browse/HBASE-15646
>
> Matteo
>
>
> On Wed, May 4, 2016 at 8:53 PM, Lex Toumbourou <lex@scrunch.com> wrote:
>
> > Hi all,
> >
> > I'm having a couple of problems with exporting HBase snapshots to S3. I
> am
> > running HBase version 1.2.0.
> >
> > I have a table called "domain"
> >
> > And I have created a snapshot for it:
> >
> > hbase(main):003:0> snapshot 'domain', 'domain-aws-test'
> > 0 row(s) in 0.3310 seconds
> >
> > ---
> >
> > I am attempting to export it to S3 using the following command:
> >
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > "domain-aws-test" -copy-to s3://my-hbase-snapshots/domain-aws-test
> >
> > Now, when I view the snapshot metadata in the S3 bucket, there's nothing
> > there:
> >
> > > aws s3 ls my-hbase-snapshots/domain-aws-snapshots
> >
> > But there is data under:
> >
> > > aws s3 ls my-hbase-snapshots/\/domain-aws-test/
> >                            PRE .hbase-snapshot/
> > 2016-05-05 13:38:12          1 .hbase-snapshot
> >
> > It seems what's happening is, HBase is creating a directory/prefix with
> no
> > name and placing the snapshot data under there.
> >
> > That wouldn't be a problem, except that when I try to import the snapshot
> > on my destination cluster, it seems unable to deal with the empty
> > directory:
> >
> > sudo -u hbase hbase snapshot export -D
> > hbase.rootdir=s3://my-hbase-snapshots/\/domain-aws-test -snapshot
> > my-aws-test -copy-to hdfs://hbaseClusterDNSName:8020/user/hbase -mappers
> 2
> >
> > Caused by: java.io.FileNotFoundException: No such file or directory
> > 's3://my-hbase-snapshots/domain-aws-test/.hbase-snapshot/domain-aws-test'
> >
> > ---
> >
> > Any one come across this before?
> >
> > Lex
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message