Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=P7t0ITyeKLSjHV44MbdL65N2i5xQ2/KPzlE7xM/6a7nbzG0FQrwwd0CznZPYJRFzKPYn97TlBc94ZimpHvuJYQOZd5iBzoa14vFp/USUS3dra2Hig5ch/2qOyarvVJo/LhKzLqRlbOXrII6ZLKNR5HhhAr4GYf7jxdFclGd5310=;
References: 
 <B5A72F6BFEB14B4CB4295E25D114BD12027E6821978E@nambx03.corp.adobe.com>
 <CADcMMgFxqs5NkF-SFaVTAU0-tnzOw7vHsxC-mS3pvd_vWoJUqA@mail.gmail.com>
 <B5A72F6BFEB14B4CB4295E25D114BD12027E68219790@nambx03.corp.adobe.com>
 <CADcMMgFwEC1Pqkm95bh72tZ+KUfPcXXnFAZUGDDFRkz9qJTmmg@mail.gmail.com>
 <CAO83RbWLpOV6MnnPeDgENOS48TkG+FyUz1=dMY_fPTJuU9-j=g@mail.gmail.com>
Message-ID: <1330068462.94620.YahooMailNeo@web121706.mail.ne1.yahoo.com>
Date: Thu, 23 Feb 2012 23:27:42 -0800 (PST)
From: lars hofhansl <lhofhansl@yahoo.com>
Reply-To: lars hofhansl <lhofhansl@yahoo.com>
Subject: Re: export/import for backup
To: "user@hbase.apache.org" <user@hbase.apache.org>
In-Reply-To: 
 <CAO83RbWLpOV6MnnPeDgENOS48TkG+FyUz1=dMY_fPTJuU9-j=g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Recovery from these exported HFiles should be extremely fast.=0A=0AWe can a=
dd an option to Export that export to HFiles instead using HFileOutputForma=
t instead.=0ANote that you will eat the sort CPU at export time, though, wh=
ich is (hopefully) more frequent than importing,=0Anot entirely sure that t=
his is a good trade-off.=0A=0A-- Lars=0A=0A________________________________=
=0AFrom: Jacques <whshub@gmail.com>=0ATo: user@hbase.apache.org =0ASent: Tu=
esday, February 21, 2012 8:22 AM=0ASubject: Re: export/import for backup=0A=
=0AI was thinking about this and have a couple thoughts...=0A=0AWhile Stack=
's solution above would work, it means a couple things: 1. if=0Ayou haven't=
 saved splits, your going to have to figure out how to pre-split=0Afor a fu=
ll restore.=A0 2. you have to wait for the data re-sort at recovery=0Atime =
instead of backup time so recovery time will be substantially longer.=0A=0A=
It seems like we should make a new script like export that automatically=0A=
exports the data as bulk importable along with all of the table's schema=0A=
and split information.=A0 We then could make an import script that simply=
=0Acreates the backed up table (to potentially a different target name) and=
=0Athen bulk imports it, pre-splitting using the splits defined on export.=
=0A(We actually did something like this recently to migrate data from one=
=0Aformat to another.)=0A=0AIt wouldn't work for the case where you are try=
ing to do a merged restore=0A(e.g. pre-existing table) but it seems like re=
covery would be really quick.=0AI suppose you could allow it to support imp=
orting into an existing table=0Abut then you may have to wait for splits on=
 a bunch of the files (I know=0Athe bulk import script is designed to do th=
is but i'm not sure how it would=0Ahandle a large amount of splits if your =
target table has diverged=0Asubstantially from when the backup was done).=
=0A=0AJacques=0A=0A=0AOn Mon, Feb 20, 2012 at 9:19 PM, Stack <stack@duboce.=
net> wrote:=0A=0A> On Mon, Feb 20, 2012 at 1:58 PM, Paul Mackles <pmackles@=
adobe.com> wrote:=0A> > Actually, an hbase export to "bulk load" facility s=
ounds like a great=0A> idea. We have been using bulk loads to migrate data =
from an older data=0A> store and they have worked awesome for us. It also d=
oesn't seem like it=0A> would be that hard to implement. So what am I missi=
ng?=0A> >=0A>=0A> Little?=0A>=0A> Check out the Import.java in mapreduce pa=
ckage.=A0 See how its pulling=0A> from SequenceFiles into a map that output=
s to a TableOutputFormat=0A> inside in the map.=A0 Make a new MR job that h=
as same input but that=0A> outputs to HFileOutputFormat instead (you'll nee=
d the total order=0A> partitioner and a reducer in the mix which Import doe=
sn't have).=0A>=0A> St.Ack=0A>