hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From oc tsdb <oc.t...@gmail.com>
Subject Re: HBase 0.94.10 snapshot FileNotFoundException exception
Date Tue, 10 Dec 2013 10:46:37 GMT
Thanks for confirming the issue.So, to overcome this issue we need to move
from 0.94.10 to either 0.94.12 or above.

Thanks
-OC


On Tue, Dec 10, 2013 at 4:09 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>wrote:

> Hi,  thank you for the follow up.
>
> The problem is related to a bug in the name resolution of a "clone" (that
> in your case the is the restore of a snapshot of a restored table).
> This problem was fixed as part of HBASE-8760, which should be integrated in
> 0.94.12
>
>
> Matteo
>
>
>
> On Tue, Dec 10, 2013 at 10:00 AM, oc tsdb <oc.tsdb@gmail.com> wrote:
>
> > Hi,
> >
> > Thanks for your help.
> >
> > More details are added below.Please let us know if any additional logs
> > required.
> >
> > We have only one cluster (cluster-1) with 1 NN and with 4 DNs and HBase
> > version is 0.94.10
> >
> > We are using hbase shell to Create, Export snapshots.
> >
> > In short, what we are trying to do is:
> >
> > -Create snapshots for all HBase tables on cluster-1 and exporte to local
> > file system.
> >    Here exported data contains archive+hbase-snapshot
> > -Remove everything from Cluster-1 and importe all exported snapshots from
> > local file system to same cluster-1.
> >  Working fine without any issues.
> > -Again,after restoring snapshots, create snapshots for all HBase tables
> on
> > cluster-1 and exporte to local file system
> >   Exported data contains only hbase-snapshot but not archive directory.
> >   As we don't have archive directory, snapshot restore is failing with
> > FileNotFoundException.
> >
> > Basically snapshot export on restored snapshot data is not working in our
> > case.
> >
> > Detailed Steps are given below:
> >
> > 1.Created snapshots for all HBase tables on  cluster-1
> >
> > snapshot 'tsdb', 'tsdb_snap_backup'
> > snapshot 'tsdb-meta', 'tsdb-meta_snap_backup'
> > snapshot 'tsdb-tree', 'tsdb-tree_snap_backup'
> > snapshot 'tsdb-uid', 'tsdb-uid_snap_backup'
> >
> > 2.Exported above snapshots to same cluster(cluster-1) under hdfs root
> > (/tsdb_full_backup_2013-12-09_23.32.19)
> >
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > tsdb_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers
> > 16;
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> >  tsdb-meta_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19
> > -mappers 16;
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> >  tsdb-tree_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19
> > -mappers 16;
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> >  tsdb-uid_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19
> > -mappers 16;
> >
> >
> > 3. Copied /tsdb_full_backup_2013-12-09_23.32.19 to local file system
> >
> > /opt/airwave/lib/hadoop/bin/hadoop fs -get
> > /tsdb_full_backup_2013-12-09_23.32.19 /tmp/
> >
> > We could see archive+hbase-snapshot directories copied to local file
> system
> > /tmp/tsdb_full_backup_2013-12-09_23.32.19
> >
> > 4. Formatted cluster-1
> > Remove everything from Cluster-1.Now its fresh cluster.
> >
> > 5. copied /tmp/tsdb_full_backup_2013-12-09_23.32.19/ to hdfs path (root
> > dir)
> >
> > /opt/airwave/lib/hadoop/bin/hadoop fs -put
> > /tmp/tsdb_full_backup_2013-12-09_23.32.19 /
> >
> > 6.Import snapshots from hdfs path /tsdb_full_backup_2013-12-09_23.32.19
> >
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D
> > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot
> > tsdb_snap_backup -copy-to hdfs:/hbase
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D
> > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot
> > tsdb-meta_snap_backup -copy-to hdfs:/hbase
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D
> > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot
> > tsdb-tree_snap_backup -copy-to hdfs:/hbase
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D
> > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot
> > tsdb-uid_snap_backup -copy-to hdfs:/hbase
> >
> > 7.disable all HBase tables
> > disable 'tsdb'
> > disable'tsdb-meta'
> > disable 'tsdb-tree'
> > disable 'tsdb-uid'
> >
> > 8.restore all HBase tables from snapshot
> >
> >
> > restore_snapshot 'tsdb'
> > restore_snapshot 'tsdb-meta'
> > restore_snapshot 'tsdb-tree'
> > restore_snapshot 'tsdb-uid'
> >
> > 9.enable all HBase tables
> > enable 'tsdb'
> > enable'tsdb-meta'
> > enable 'tsdb-tree'
> > enable 'tsdb-uid'
> >
> > 10.Deleted all snapshots
> >
> > 11.Verified HBase data is imported properly or not.
> >   data is imported properly.No issues found.
> >
> > 12.Now again repeat step 1 to step 11 on imported data.
> >
> >    In this case hbase-snapshot directory only exported to local file
> system
> > (eg : /tmp/tsdb_full_backup_2013-12-09_23.42.19)
> >    But we dont see archive directory in local file system (eg :
> > /tmp/tsdb_full_backup_2013-12-09_23.42.19) why?
> >
> > Thanks
> > -OC
> >
> >
> > On Mon, Dec 9, 2013 at 5:08 PM, Matteo Bertozzi <theo.bertozzi@gmail.com
> > >wrote:
> >
> > > can you clarify your steps, maybe adding the commands that you used?
> > > Which hbase version are you using? (HBASE-8455 doesn't seems to be
> > related
> > > from the log)
> > >
> > > I assume:
> > > 1) Create, Export and Verify snapshot means
> > >  - from hbase shell/client code on Cluster-1 take the snapshot:
> snapshot
> > > 'table_name', 'snapshotName'
> > >  - from a shell of Cluster-1 export snapshot on Cluster-2 with: hbase
> > > org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshotName
> > ...
> > >  - from a shell on Cluster-2 verify snapshot with: hbase
> > > org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName
> > > 2) Started HBase from scratch means
> > >  - Remove everything from Cluster 1 with: disable 'tableName' .. drop
> > > 'tableName' .. delete_snapshot 'snapshotName'
> > > 3) Import snapshot created in Step 1 means
> > >  - from a shell in Cluster-2 export snapshot to Cluster-1: hbase
> > > org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshotName
> > ...
> > >  - from a shell on Cluster-1 verify snapshot with: hbase
> > > org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName
> > >  - from a shell on Cluster-1: clone_snapshot 'snapshotName',
> 'tableName'
> > > 4 and 5 I've no idea what you mean
> > >
> > > Matteo
> > >
> > >
> > >
> > > On Mon, Dec 9, 2013 at 11:25 AM, oc tsdb <oc.tsdb@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > we are taking tsdb data backup using HBase snapshots.
> > > >
> > > > We tested the scenario as mentioned below:
> > > >
> > > > 1. Created snapshots and exported all snapshots successfully.
> > > >     Exported data contains archive+hbase-snapshot
> > > > 2. Started HBase from scratch (removed all Hbase tabled and created
> > > again)
> > > > 3. Successfully imported all snapshots created in step 1.
> > > > 4. Again Created snapshots on imported data (step 3) and exported all
> > > > snapshots.
> > > >      Exported data contains only hbase-snapshot but not archive.Why?
> > > > 5. Now tried to import snapshots created in step 4 but we are getting
> > > below
> > > > exception.
> > > >
> > > > Exception in thread "main" java.io.FileNotFoundException: Unable to
> > open
> > > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
> > > >
> > > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e
> > > > ,
> > > > hdfs://
> > > >
> > > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e
> > > > ,
> > > > hdfs://
> > > >
> > > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e
> > > > ]
> > > > at
> org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> > > >
> > > >
> > > > Exception in thread "main" java.io.FileNotFoundException: Unable to
> > open
> > > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
> > > >
> > > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c
> > > > ,
> > > > hdfs://
> > > >
> > > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c
> > > > ,
> > > > hdfs://
> > > >
> > > >
> > >
> >
> tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c
> > > > ]
> > > > at
> org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> > > >
> > > >
> > > > Can someone help us to understand the reason to fail snapshot import.
> > > > Is this issue similar to HBASE-8455?
> > > >
> > > > Thanks
> > > > -OC
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message