Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com
 designates 209.85.220.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAH37JUT=ft4RuptLTWwAEtWS=WCMTKKQR4Qn05qcGVbsaO6Udw@mail.gmail.com>
References: 
 <CAH37JUShuPEvVHfeWTihcdL5Kd_fT6tUWZzxzOguK6LcEdxX+g@mail.gmail.com>
	<CALte62y5+hgR=WyFjRTb-a6_1d5sV_5S3bdyZr9oj03G1b3Dtg@mail.gmail.com>
	<CAH37JURut2+uPfHKm72JOLvmy59MZ=bbMZDrmfiKNWbwQ-Risg@mail.gmail.com>
	<CALte62z+tdud2A8G+hsGXdKuapRgM1u1z-OFzoCHkD2_iUKedg@mail.gmail.com>
	<CAH37JUT=ft4RuptLTWwAEtWS=WCMTKKQR4Qn05qcGVbsaO6Udw@mail.gmail.com>
Date: Tue, 3 Dec 2013 04:09:32 -0800
Message-ID: 
 <CALte62ww1NF9N2mjqc38j4b0jiWmXy4o15aF=VO2OGbA0KSOzQ@mail.gmail.com>
Subject: Re: HBase ExportSnapshot
From: Ted Yu <yuzhihong@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7bea3f6450f63104eca02bf1

--047d7bea3f6450f63104eca02bf1
Content-Type: text/plain; charset=ISO-8859-1

Please check what happened to the HFileLink mentioned in the exception
below - this would be the cause for snapshot export failure:

Exception in thread "main" java.io.FileNotFoundException: Unable to open
link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
,
hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
,
hdfs://
site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
]
        at
org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)


On Mon, Dec 2, 2013 at 10:01 PM, oc tsdb <oc.tsdb@gmail.com> wrote:

> here is snapshot export logs.
>
> mastre log:
> ===========
> 2013-12-02 21:54:30,840 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
>
> snapshot export console log:
> =============================
>
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
> 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=1 regions=1
> average=1.0 mostloaded=1 leastloaded=1
>
>         at
>
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
>         at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
>         at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
>         at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
> 13/12/02 21:54:24 INFO util.FSVisitor: No families under region
> directory:hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/f06335933b32019c4369f95001d996fb
> 13/12/02 21:54:24 INFO util.FSVisitor: No logs under directory:hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/.logs
> 13/12/02 21:54:24 WARN snapshot.ExportSnapshot: There are 0 store file to
> be copied. There may be no data in the table.
> 13/12/02 21:54:25 INFO util.FSVisitor: No families under region
> directory:hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/c40c34c4312ccb3302fbaf62caa91b9c
> 13/12/02 21:54:25 INFO util.FSVisitor: No logs under directory:hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/.logs
> 13/12/02 21:54:25 WARN snapshot.ExportSnapshot: There are 0 store file to
> be copied. There may be no data in the table.
> Exception in thread "main" java.io.FileNotFoundException: Unable to open
> link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ,
> hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ,
> hdfs://
>
> site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645
> ]
>         at
> org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376)
>         at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390)
>         at
>
> org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
>         at
>
> org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81)
>         at
>
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116)
>         at
>
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101)
>         at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385)
>         at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705)
>         at
>
> org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709)
>
> Basically while exporting to local i dont see .archive directory.why?
>
> Please comment on this -
> It seems we can't export complete snapshot data directly to local file
> system using 'ExportSnapshot' command.
> If we want to copy to outside of cluster first we need to export it to hdfs
> and then use hadoop get command to copy to local file system.
> Is this correct?
>
> Thanks
> -OC
>
>
> On Tue, Dec 3, 2013 at 11:09 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > The log you pasted was for taking snapshot.
> > Do you have log from ExportSnapshot ?
> >
> > bq. What is the difference between below two commands?
> >
> > This is the code that determines output FileSystem:
> >
> >     FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf);
> >
> > For 'file:///tmp/hbase_backup' argument, outputFs would be an instance of
> > org.apache.hadoop.fs.LocalFileSystem.
> >
> > Cheers
> >
> >
> > On Mon, Dec 2, 2013 at 9:06 PM, oc tsdb <oc.tsdb@gmail.com> wrote:
> >
> > > We see same logs for both options
> > >
> > > 013-12-02 09:47:41,311 INFO
> > > org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running
> > FLUSH
> > > table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb
> > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > > FileSystem doesn't support getDefaultReplication
> > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils:
> > > FileSystem doesn't support getDefaultBlockSize
> > > 2013-12-02 09:47:41,337 INFO
> org.apache.hadoop.hbase.procedure.Procedure:
> > > Starting procedure 'tsdb_snap_backup'
> > > 2013-12-02 09:47:41,724 INFO
> org.apache.hadoop.hbase.procedure.Procedure:
> > > Procedure 'tsdb_snap_backup' execution completed
> > > 2013-12-02 09:47:41,724 INFO
> > > org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes
> > for
> > > procedure tsdb_snap_backupincluding nodes
> /hbase/online-snapshot/acquired
> > > /hbase/online-snapshot/reached /hbase/online-snapshot/abort
> > > 2013-12-02 09:47:41,730 INFO
> > > org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler:
> Done
> > > waiting - snapshot for tsdb_snap_backup finished!
> > >
> > > It seems we can't export complete snapshot data directly to local file
> > > system using 'ExportSnapshot' command.
> > > If we want to copy to outside of cluster first we need to export it to
> > hdfs
> > > and then use hadoop get command to copy to local file system.
> > > Is this correct?
> > >
> > > What is the difference between below two commands?
> > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > >  hbase_tbl_snapshot_name -copy-to   file:///tmp/hbase_backup -mappers
> 16;
> > >
> > >  hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > >  hbase_tbl_snapshot_name -copy-to   hdfs:/hbase_backup -mappers 16;
> > >
> > > Thanks
> > > -OC
> > >
> > >
> > >
> > > On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > Can you pastebin master log during operation #2 ?
> > > >
> > > > There have been at least two fixes since 0.94.10, listed below.
> > > > It would be nice if you can verify this behavior using 0.94.14
> > > >
> > > > Cheers
> > > >
> > > > r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013)
> | 1
> > > > line
> > > >
> > > > HBASE-8760 possible loss of data in snapshot taken after region split
> > > >
> > ------------------------------------------------------------------------
> > > > r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013)
> | 1
> > > > line
> > > >
> > > > HBASE-9060 ExportSnapshot job fails if target path contains
> percentage
> > > > character (Jerry He)
> > > >
> > > >
> > > > On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb <oc.tsdb@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We have cluster with 4 data nodes and HBase version is 0.94.10.
> > > > >
> > > > > We have created snapshot for all hbase tables and trying to export
> > > > snapshot
> > > > > in two ways.
> > > > >
> > > > > option 1.Export snapshot into same cluster hdfs
> > > > >
> > > > >  hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > >  hbase_tbl_snapshot_name -copy-to   *hdfs:/hbase_backup *-mappers
> 16;
> > > > >
> > > > > Here we are getting full data ( .archive + .hbase-snapshot)
> exported
> > to
> > > > > hdfs:/hbase_backup
> > > > >
> > > > > option 2.Export snapshot to local filesystem
> > > > > command :
> > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > > >  hbase_tbl_snapshot_name -copy-to   *file:///tmp/hbase_backup*
> > -mappers
> > > > 16;
> > > > >
> > > > > But with option 2 we only getting .hbase-snapshot exported to local
> > dir
> > > > > (/tmp/hbase_backup) but .archive files are not exported.It is
> > expected
> > > > > behavior or something wrong in option 2.
> > > > >
> > > > > Thanks
> > > > > OC
> > > > >
> > > >
> > >
> >
>

--047d7bea3f6450f63104eca02bf1--