hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matteo Bertozzi <theo.berto...@gmail.com>
Subject Re: HBase 0.94.10 snapshot FileNotFoundException exception
Date Tue, 10 Dec 2013 10:39:44 GMT
Hi,  thank you for the follow up.

The problem is related to a bug in the name resolution of a "clone" (that
in your case the is the restore of a snapshot of a restored table).
This problem was fixed as part of HBASE-8760, which should be integrated in
0.94.12


Matteo



On Tue, Dec 10, 2013 at 10:00 AM, oc tsdb <oc.tsdb@gmail.com> wrote:

> Hi,
>
> Thanks for your help.
>
> More details are added below.Please let us know if any additional logs
> required.
>
> We have only one cluster (cluster-1) with 1 NN and with 4 DNs and HBase
> version is 0.94.10
>
> We are using hbase shell to Create, Export snapshots.
>
> In short, what we are trying to do is:
>
> -Create snapshots for all HBase tables on cluster-1 and exporte to local
> file system.
>    Here exported data contains archive+hbase-snapshot
> -Remove everything from Cluster-1 and importe all exported snapshots from
> local file system to same cluster-1.
>  Working fine without any issues.
> -Again,after restoring snapshots, create snapshots for all HBase tables on
> cluster-1 and exporte to local file system
>   Exported data contains only hbase-snapshot but not archive directory.
>   As we don't have archive directory, snapshot restore is failing with
> FileNotFoundException.
>
> Basically snapshot export on restored snapshot data is not working in our
> case.
>
> Detailed Steps are given below:
>
> 1.Created snapshots for all HBase tables on  cluster-1
>
> snapshot 'tsdb', 'tsdb_snap_backup'
> snapshot 'tsdb-meta', 'tsdb-meta_snap_backup'
> snapshot 'tsdb-tree', 'tsdb-tree_snap_backup'
> snapshot 'tsdb-uid', 'tsdb-uid_snap_backup'
>
> 2.Exported above snapshots to same cluster(cluster-1) under hdfs root
> (/tsdb_full_backup_2013-12-09_23.32.19)
>
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> tsdb_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers
> 16;
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
>  tsdb-meta_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19
> -mappers 16;
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
>  tsdb-tree_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19
> -mappers 16;
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
>  tsdb-uid_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19
> -mappers 16;
>
>
> 3. Copied /tsdb_full_backup_2013-12-09_23.32.19 to local file system
>
> /opt/airwave/lib/hadoop/bin/hadoop fs -get
> /tsdb_full_backup_2013-12-09_23.32.19 /tmp/
>
> We could see archive+hbase-snapshot directories copied to local file system
> /tmp/tsdb_full_backup_2013-12-09_23.32.19
>
> 4. Formatted cluster-1
> Remove everything from Cluster-1.Now its fresh cluster.
>
> 5. copied /tmp/tsdb_full_backup_2013-12-09_23.32.19/ to hdfs path (root
> dir)
>
> /opt/airwave/lib/hadoop/bin/hadoop fs -put
> /tmp/tsdb_full_backup_2013-12-09_23.32.19 /
>
> 6.Import snapshots from hdfs path /tsdb_full_backup_2013-12-09_23.32.19
>
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D
> hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot
> tsdb_snap_backup -copy-to hdfs:/hbase
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D
> hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot
> tsdb-meta_snap_backup -copy-to hdfs:/hbase
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D
> hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot
> tsdb-tree_snap_backup -copy-to hdfs:/hbase
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D
> hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot
> tsdb-uid_snap_backup -copy-to hdfs:/hbase
>
> 7.disable all HBase tables
> disable 'tsdb'
> disable'tsdb-meta'
> disable 'tsdb-tree'
> disable 'tsdb-uid'
>
> 8.restore all HBase tables from snapshot
>
>
> restore_snapshot 'tsdb'
> restore_snapshot 'tsdb-meta'
> restore_snapshot 'tsdb-tree'
> restore_snapshot 'tsdb-uid'
>
> 9.enable all HBase tables
> enable 'tsdb'
> enable'tsdb-meta'
> enable 'tsdb-tree'
> enable 'tsdb-uid'
>
> 10.Deleted all snapshots
>
> 11.Verified HBase data is imported properly or not.
>   data is imported properly.No issues found.
>
> 12.Now again repeat step 1 to step 11 on imported data.
>
>    In this case hbase-snapshot directory only exported to local file system
> (eg : /tmp/tsdb_full_backup_2013-12-09_23.42.19)
>    But we dont see archive directory in local file system (eg :
> /tmp/tsdb_full_backup_2013-12-09_23.42.19) why?
>
> Thanks
> -OC
>
>
> On Mon, Dec 9, 2013 at 5:08 PM, Matteo Bertozzi <theo.bertozzi@gmail.com
> >wrote:
>
> > can you clarify your steps, maybe adding the commands that you used?
> > Which hbase version are you using? (HBASE-8455 doesn't seems to be
> related
> > from the log)
> >
> > I assume:
> > 1) Create, Export and Verify snapshot means
> >  - from hbase shell/client code on Cluster-1 take the snapshot: snapshot
> > 'table_name', 'snapshotName'
> >  - from a shell of Cluster-1 export snapshot on Cluster-2 with: hbase
> > org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshotName
> ...
> >  - from a shell on Cluster-2 verify snapshot with: hbase
> > org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName
> > 2) Started HBase from scratch means
> >  - Remove everything from Cluster 1 with: disable 'tableName' .. drop
> > 'tableName' .. delete_snapshot 'snapshotName'
> > 3) Import snapshot created in Step 1 means
> >  - from a shell in Cluster-2 export snapshot to Cluster-1: hbase
> > org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshotName
> ...
> >  - from a shell on Cluster-1 verify snapshot with: hbase
> > org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName
> >  - from a shell on Cluster-1: clone_snapshot 'snapshotName', 'tableName'
> > 4 and 5 I've no idea what you mean
> >
> > Matteo
> >
> >
> >
> > On Mon, Dec 9, 2013 at 11:25 AM, oc tsdb <oc.tsdb@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > we are taking tsdb data backup using HBase snapshots.
> > >
> > > We tested the scenario as mentioned below:
> > >
> > > 1. Created snapshots and exported all snapshots successfully.
> > >     Exported data contains archive+hbase-snapshot
> > > 2. Started HBase from scratch (removed all Hbase tabled and created
> > again)
> > > 3. Successfully imported all snapshots created in step 1.
> > > 4. Again Created snapshots on imported data (step 3) and exported all
> > > snapshots.
> > >      Exported data contains only hbase-snapshot but not archive.Why?
> > > 5. Now tried to import snapshots created in step 4 but we are getting
> > below
> > > exception.
> > >
> > > Exception in thread "main" java.io.FileNotFoundException: Unable to
> open
> > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
> > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e
> > > ,
> > > hdfs://
> > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e
> > > ,
> > > hdfs://
> > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e
> > > ]
> > > at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> > >
> > >
> > > Exception in thread "main" java.io.FileNotFoundException: Unable to
> open
> > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
> > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c
> > > ,
> > > hdfs://
> > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c
> > > ,
> > > hdfs://
> > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c
> > > ]
> > > at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> > >
> > >
> > > Can someone help us to understand the reason to fail snapshot import.
> > > Is this issue similar to HBASE-8455?
> > >
> > > Thanks
> > > -OC
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message