hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From oc tsdb <oc.t...@gmail.com>
Subject Re: HBase ExportSnapshot
Date Tue, 03 Dec 2013 06:01:25 GMT
here is snapshot export logs.

mastre log:
===========
2013-12-02 21:54:30,840 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1

snapshot export console log:
=============================

2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1
2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=1
average=1.0 mostloaded=1 leastloaded=1

        at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
        at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
        at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
        at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
13/12/02 21:54:24 INFO util.FSVisitor: No families under region
directory:hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/f06335933b32019c4369f95001d996fb
13/12/02 21:54:24 INFO util.FSVisitor: No logs under directory:hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/.logs
13/12/02 21:54:24 WARN snapshot.ExportSnapshot: There are 0 store file to
be copied. There may be no data in the table.
13/12/02 21:54:25 INFO util.FSVisitor: No families under region
directory:hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/c40c34c4312ccb3302fbaf62caa91b9c
13/12/02 21:54:25 INFO util.FSVisitor: No logs under directory:hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/.logs
13/12/02 21:54:25 WARN snapshot.ExportSnapshot: There are 0 store file to
be copied. There may be no data in the table.
Exception in thread "main" java.io.FileNotFoundException: Unable to open
link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645,
hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645,
hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
]
        at
org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
        at
org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
        at
org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
        at
org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
        at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
        at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
        at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
        at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
        at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)

Basically while exporting to local i dont see .archive directory.why?

Please comment on this -
It seems we can't export complete snapshot data directly to local file
system using 'ExportSnapshot' command.
If we want to copy to outside of cluster first we need to export it to hdfs
and then use hadoop get command to copy to local file system.
Is this correct?

Thanks
-OC


On Tue, Dec 3, 2013 at 11:09 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> The log you pasted was for taking snapshot.
> Do you have log from ExportSnapshot ?
>
> bq. What is the difference between below two commands?
>
> This is the code that determines output FileSystem:
>
>     FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf);
>
> For 'file:///tmp/hbase_backup' argument, outputFs would be an instance of
> org.apache.hadoop.fs.LocalFileSystem.
>
> Cheers
>
>
> On Mon, Dec 2, 2013 at 9:06 PM, oc tsdb <oc.tsdb@gmail.com> wrote:
>
> > We see same logs for both options
> >
> > 013-12-02 09:47:41,311 INFO
> > org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running
> FLUSH
> > table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb
> > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > FileSystem doesn't support getDefaultReplication
> > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > FileSystem doesn't support getDefaultBlockSize
> > 2013-12-02 09:47:41,337 INFO org.apache.hadoop.hbase.procedure.Procedure:
> > Starting procedure 'tsdb_snap_backup'
> > 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.Procedure:
> > Procedure 'tsdb_snap_backup' execution completed
> > 2013-12-02 09:47:41,724 INFO
> > org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes
> for
> > procedure tsdb_snap_backupincluding nodes /hbase/online-snapshot/acquired
> > /hbase/online-snapshot/reached /hbase/online-snapshot/abort
> > 2013-12-02 09:47:41,730 INFO
> > org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler: Done
> > waiting - snapshot for tsdb_snap_backup finished!
> >
> > It seems we can't export complete snapshot data directly to local file
> > system using 'ExportSnapshot' command.
> > If we want to copy to outside of cluster first we need to export it to
> hdfs
> > and then use hadoop get command to copy to local file system.
> > Is this correct?
> >
> > What is the difference between below two commands?
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> >  hbase_tbl_snapshot_name -copy-to   file:///tmp/hbase_backup -mappers 16;
> >
> >  hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> >  hbase_tbl_snapshot_name -copy-to   hdfs:/hbase_backup -mappers 16;
> >
> > Thanks
> > -OC
> >
> >
> >
> > On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Can you pastebin master log during operation #2 ?
> > >
> > > There have been at least two fixes since 0.94.10, listed below.
> > > It would be nice if you can verify this behavior using 0.94.14
> > >
> > > Cheers
> > >
> > > r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013) | 1
> > > line
> > >
> > > HBASE-8760 possible loss of data in snapshot taken after region split
> > >
> ------------------------------------------------------------------------
> > > r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013) | 1
> > > line
> > >
> > > HBASE-9060 ExportSnapshot job fails if target path contains percentage
> > > character (Jerry He)
> > >
> > >
> > > On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb <oc.tsdb@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > We have cluster with 4 data nodes and HBase version is 0.94.10.
> > > >
> > > > We have created snapshot for all hbase tables and trying to export
> > > snapshot
> > > > in two ways.
> > > >
> > > > option 1.Export snapshot into same cluster hdfs
> > > >
> > > >  hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > >  hbase_tbl_snapshot_name -copy-to   *hdfs:/hbase_backup *-mappers 16;
> > > >
> > > > Here we are getting full data ( .archive + .hbase-snapshot) exported
> to
> > > > hdfs:/hbase_backup
> > > >
> > > > option 2.Export snapshot to local filesystem
> > > > command :
> > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > >  hbase_tbl_snapshot_name -copy-to   *file:///tmp/hbase_backup*
> -mappers
> > > 16;
> > > >
> > > > But with option 2 we only getting .hbase-snapshot exported to local
> dir
> > > > (/tmp/hbase_backup) but .archive files are not exported.It is
> expected
> > > > behavior or something wrong in option 2.
> > > >
> > > > Thanks
> > > > OC
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message