hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From oc tsdb <oc.t...@gmail.com>
Subject Re: HBase ExportSnapshot
Date Tue, 03 Dec 2013 15:26:49 GMT
I think that error is  because of missing .archive files?

We only see this issue when we export snapshot to local file system
(file:///tmp/hbase_backup)
but not when we export snapshot to hdfs.

When we export snapshot to hdfs we are getting all the data.Its only not
working when we export snapshot data to local file system directly?



On Tue, Dec 3, 2013 at 5:39 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Please check what happened to the HFileLink mentioned in the exception
> below - this would be the cause for snapshot export failure:
>
> Exception in thread "main" java.io.FileNotFoundException: Unable to open
> link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ,
> hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ,
> hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ]
>         at
> org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
>
>
> On Mon, Dec 2, 2013 at 10:01 PM, oc tsdb <oc.tsdb@gmail.com> wrote:
>
> > here is snapshot export logs.
> >
> > mastre log:
> > ===========
> > 2013-12-02 21:54:30,840 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> >
> > snapshot export console log:
> > =============================
> >
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> > Skipping load balancing because balanced cluster; servers=1 regions=1
> > average=1.0 mostloaded=1 leastloaded=1
> >
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> >         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> > 13/12/02 21:54:24 INFO util.FSVisitor: No families under region
> > directory:hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/f06335933b32019c4369f95001d996fb
> > 13/12/02 21:54:24 INFO util.FSVisitor: No logs under directory:hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/.logs
> > 13/12/02 21:54:24 WARN snapshot.ExportSnapshot: There are 0 store file to
> > be copied. There may be no data in the table.
> > 13/12/02 21:54:25 INFO util.FSVisitor: No families under region
> > directory:hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/c40c34c4312ccb3302fbaf62caa91b9c
> > 13/12/02 21:54:25 INFO util.FSVisitor: No logs under directory:hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/.logs
> > 13/12/02 21:54:25 WARN snapshot.ExportSnapshot: There are 0 store file to
> > be copied. There may be no data in the table.
> > Exception in thread "main" java.io.FileNotFoundException: Unable to open
> > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> > ,
> > hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> > ,
> > hdfs://
> >
> >
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> > ]
> >         at
> > org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
> >         at
> >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
> >         at
> >
> >
> org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
> >         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
> >         at
> >
> >
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> >
> > Basically while exporting to local i dont see .archive directory.why?
> >
> > Please comment on this -
> > It seems we can't export complete snapshot data directly to local file
> > system using 'ExportSnapshot' command.
> > If we want to copy to outside of cluster first we need to export it to
> hdfs
> > and then use hadoop get command to copy to local file system.
> > Is this correct?
> >
> > Thanks
> > -OC
> >
> >
> > On Tue, Dec 3, 2013 at 11:09 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > The log you pasted was for taking snapshot.
> > > Do you have log from ExportSnapshot ?
> > >
> > > bq. What is the difference between below two commands?
> > >
> > > This is the code that determines output FileSystem:
> > >
> > >     FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf);
> > >
> > > For 'file:///tmp/hbase_backup' argument, outputFs would be an instance
> of
> > > org.apache.hadoop.fs.LocalFileSystem.
> > >
> > > Cheers
> > >
> > >
> > > On Mon, Dec 2, 2013 at 9:06 PM, oc tsdb <oc.tsdb@gmail.com> wrote:
> > >
> > > > We see same logs for both options
> > > >
> > > > 013-12-02 09:47:41,311 INFO
> > > > org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running
> > > FLUSH
> > > > table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb
> > > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > > > FileSystem doesn't support getDefaultReplication
> > > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > > > FileSystem doesn't support getDefaultBlockSize
> > > > 2013-12-02 09:47:41,337 INFO
> > org.apache.hadoop.hbase.procedure.Procedure:
> > > > Starting procedure 'tsdb_snap_backup'
> > > > 2013-12-02 09:47:41,724 INFO
> > org.apache.hadoop.hbase.procedure.Procedure:
> > > > Procedure 'tsdb_snap_backup' execution completed
> > > > 2013-12-02 09:47:41,724 INFO
> > > > org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all
> znodes
> > > for
> > > > procedure tsdb_snap_backupincluding nodes
> > /hbase/online-snapshot/acquired
> > > > /hbase/online-snapshot/reached /hbase/online-snapshot/abort
> > > > 2013-12-02 09:47:41,730 INFO
> > > > org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler:
> > Done
> > > > waiting - snapshot for tsdb_snap_backup finished!
> > > >
> > > > It seems we can't export complete snapshot data directly to local
> file
> > > > system using 'ExportSnapshot' command.
> > > > If we want to copy to outside of cluster first we need to export it
> to
> > > hdfs
> > > > and then use hadoop get command to copy to local file system.
> > > > Is this correct?
> > > >
> > > > What is the difference between below two commands?
> > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > >  hbase_tbl_snapshot_name -copy-to   file:///tmp/hbase_backup -mappers
> > 16;
> > > >
> > > >  hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > >  hbase_tbl_snapshot_name -copy-to   hdfs:/hbase_backup -mappers 16;
> > > >
> > > > Thanks
> > > > -OC
> > > >
> > > >
> > > >
> > > > On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > > >
> > > > > Can you pastebin master log during operation #2 ?
> > > > >
> > > > > There have been at least two fixes since 0.94.10, listed below.
> > > > > It would be nice if you can verify this behavior using 0.94.14
> > > > >
> > > > > Cheers
> > > > >
> > > > > r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013)
> > | 1
> > > > > line
> > > > >
> > > > > HBASE-8760 possible loss of data in snapshot taken after region
> split
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013)
> > | 1
> > > > > line
> > > > >
> > > > > HBASE-9060 ExportSnapshot job fails if target path contains
> > percentage
> > > > > character (Jerry He)
> > > > >
> > > > >
> > > > > On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb <oc.tsdb@gmail.com>
wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We have cluster with 4 data nodes and HBase version is 0.94.10.
> > > > > >
> > > > > > We have created snapshot for all hbase tables and trying to
> export
> > > > > snapshot
> > > > > > in two ways.
> > > > > >
> > > > > > option 1.Export snapshot into same cluster hdfs
> > > > > >
> > > > > >  hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > > >  hbase_tbl_snapshot_name -copy-to   *hdfs:/hbase_backup *-mappers
> > 16;
> > > > > >
> > > > > > Here we are getting full data ( .archive + .hbase-snapshot)
> > exported
> > > to
> > > > > > hdfs:/hbase_backup
> > > > > >
> > > > > > option 2.Export snapshot to local filesystem
> > > > > > command :
> > > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > > >  hbase_tbl_snapshot_name -copy-to   *file:///tmp/hbase_backup*
> > > -mappers
> > > > > 16;
> > > > > >
> > > > > > But with option 2 we only getting .hbase-snapshot exported to
> local
> > > dir
> > > > > > (/tmp/hbase_backup) but .archive files are not exported.It is
> > > expected
> > > > > > behavior or something wrong in option 2.
> > > > > >
> > > > > > Thanks
> > > > > > OC
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message