hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sriram Muthuswamy Chittathoor" <srir...@ivycomptech.com>
Subject RE: HBase bulk load
Date Thu, 21 Jan 2010 20:35:05 GMT

The output of job 1, the one that parses the 4k files and outputs
user+day/value, if its ordered by user+day, then you can take the
outputs of
this first job and feed them to the second job one at a time.   HFiles
will
be written for some subset of all users but for this subset, all of
their
activity over the 8 years will be processed.  You'll then move on to the
next set of users....


---  I am assuming here I will mark  that a certain set of users data
(for all 8 years) goes into a certain hfile and this hfile will just
keep getting appended to for this same set of users as I progress
through different years data for the same set of users

---  I will have to completely process this set of users data year at a
time in order (2000, 2001 etc)


Maybe write this back to hdfs as sequencefiles rather than as hfiles and
then take the output of this jobs reducer and feed these to your
hfileoutputformat job one at a time if you want to piecemeal the
creation of hfiles

---  I light of the above where does this sequence files fit in ?


-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
stack
Sent: Saturday, January 16, 2010 5:58 AM
To: hbase-user@hadoop.apache.org
Subject: Re: HBase bulk load

On Thu, Jan 14, 2010 at 11:05 PM, Sriram Muthuswamy Chittathoor <
sriramc@ivycomptech.com> wrote:

>
> --- We need to bulk load 8 years worth of data from our archives.
That
> will 8 * 12 months  of data.
>

Have you run mapreduce jobs over this archive in the past?  I ask
because if
you have, you may have an input on how long it'll take to do the big or
part
of the import.



> Whats your original key made of?
> --  Each Data files is a 4K text data which has 6 players data on an
> average.  We will parse it and extract per userid/day data (so many
each
> of this would be < .5K)
>


Is your archive in HDFS now?  Are the 4k files concatenated into some
kinda
archive format?  Gzip or something?  Is it accessible with http?


>
> Would you do this step in multiple stages or feed this mapreduce job
all
> 10
> years of data?
>
>
> Either way I can do.  Since I have 8 years worth of archived data I
need
> to get them onto to the system as a one time effort.  If I proceed in
> this year order will it be fine --  2000 , 2001 , 2002.  The only
> requirement is at the end these individual years data (in hfiles)
needs
> to be loaded in Hbase.
>


If I were to guess, the data in the year 2000 is < 2001 and so on?

Is a table per year going to cut it for you?  Don't you want to see the
user
data over the whole 8 years?  It'll be a pain doing 8 different queries
and
aggregating instead of doing one query against a single table?


>
> --  Can u give me some link to doing this.   If I am getting u right
is
> this the sequence
>
> 1.  Start with say year 2000 (1 billion 4k files to be processed and
> loaded)
> 2.  Divide it into splits initially based on just filename ranges
> (user/day data is hidden inside the file)
> 3.  Each mappers gets a bunch of file (if it is 20 mappers then each
one
> will have to process 50 million 4k files (Seems too much even for a
> single year ?? --  should I go to a single month processing at a time
> ??)
>

For 50 million 4k files, you'd want more than 20 mappers.  You might
have 20
'slots' for tasks on your cluster with each time a mapper runs, it might
process N files.  1 file only would probably be too little work to
justify
the firing up of the JVM to run the task. So, you should feed each map
task
100 or 1000 4k files?  If 1k files per map task thats 50k map tasks?



> 4.  Each mapper parses the file and extract the user/day records
> 5.  The custom parttioner sends range of users/day to a particular
> reducer
>

Yes.  Your custom partitioner guarantees that a particular user only
goes to
one reducer.  How many users do you have do you think?  Maybe this job
is
too big to do all in the one go.  You need to come up with a process
with
more steps.  A MR job that runs for weeks will fail (smile).  Someone
will
for sure pull plug on the namenode just as the job is coming to an end.



> 6.  reducer in parallel will generate sequence files -- multiple will
be
> there
>
>
> My question here is in each year there will be sequence files
containing
> a range of users data.  Do I need to identify these and put them
> together in one hfile as the user/day records for all the 10 years
> should be together in the final hfile ?


No.  As stuff flows into the hfiles, it just needs to be guaranteed
ordered.
 A particular user may span multiple hfiles.


> So some manual stuff is required
> here taking related sequence files (those containing the same range of
> users / day data) and feeding them to  hfileoutputformat job ?
>


The output of job 1, the one that parses the 4k files and outputs
user+day/value, if its ordered by user+day, then you can take the
outputs of
this first job and feed them to the second job one at a time.   HFiles
will
be written for some subset of all users but for this subset, all of
their
activity over the 8 years will be processed.  You'll then move on to the
next set of users....   Eventually you will have many hfiles to upload
into
an hbase instance.  You'll need to probably modify loadtable.rb some
(One
modification you should do I thought is NOT to load an hfile whose
length is
0 bytes).




> -- - Could u also give some links to this multiput technique ??
>
>
Ryan should put up the patch soon (
https://issues.apache.org/jira/browse/HBASE-2066).


This seems like a pretty big job.  My guess is that its going to take a
bit
of time getting it all working.  Given your scale, my guess is that
you'll
run into some interesting issues.  For example, how many of those 4k
files
have corruption in them and how will your map tasks deal with the
corruption?

You need to also figure out some things like how long each step is going
to
take, how big the resultant data is going to be, and so on so you can
guage
things like the amount of hardware you are going to need to get the job
done.

The best way to get answers on the above is to start in with running a
few
mapreduce jobs passing subsets of the data to see how things work out.

Yours,
St.Ack




>
> St.Ack
>
>
>
>
> >
> > -----Original Message-----
> > From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> > stack
> > Sent: Thursday, January 14, 2010 11:33 AM
> > To: hbase-user@hadoop.apache.org
> > Subject: Re: HBase bulk load
> >
> > On Wed, Jan 13, 2010 at 9:49 PM, Sriram Muthuswamy Chittathoor <
> > sriramc@ivycomptech.com> wrote:
> >
> > > I am trying to use this technique to say bulk load 20 billion
rows.
> I
> > > tried it on a smaller set 20 million rows. A few things I had to
> take
> > > care was to write a custom partitioning logic so that a range of
> keys
> > > only go to a particular reduce since there was some mention of
> global
> > > ordering.
> > > For example  Users  (1 --  1mill) ---> Reducer 1 and so on
> > >
> > > Good.
> >
> >
> >
> > > My questions are:
> > > 1.  Can I divide the bulk loading into multiple runs  --  the
> existing
> > > bulk load bails out if it finds a HDFS output directory with the
> same
> > > name
> > >
> >
> > No.  Its not currently written to do that but especially if your
keys
> > are
> > ordered, it probably wouldn't take much to make the above work
(first
> > job
> > does the first set of keys, and so on).
> >
> >
> > > 2.  What I want to do is make multiple runs of 10 billion and then
> > > combine the output before running  loadtable.rb --  is this
possible
> ?
> > > I am thinking this may be required in case my MR bulk loading
fails
> in
> > > between and I need to start from where I crashed
> > >
> > > Well, MR does retries but, yeah, you could run into some issue at
> the
> > 10B
> > mark and want to then start over from there rather than start from
the
> > beginning.
> >
> > One thing that the current setup does not do is remove the task
hfile
> on
> > failure.  We should add this.  Would fix case where when speculative
> > execution is enabled, and the speculative tasks are kiled, we don't
> > leave
> > around half-made hfiles (Currently I believe they they show as
> > zero-length
> > files).
> >
> > St.Ack
> >
> >
> >
> > > Any tips with huge bulk loading experience ?
> > >
> > >
> > > -----Original Message-----
> > > From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf
Of
> > > stack
> > > Sent: Thursday, January 14, 2010 6:19 AM
> > > To: hbase-user@hadoop.apache.org
> > > Subject: Re: HBase bulk load
> > >
> > > See
> > >
> >
>
http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/
> > > mapreduce/package-summary.html#bulk
> > > St.Ack
> > >
> > > On Wed, Jan 13, 2010 at 4:30 PM, Ted Yu <yuzhihong@gmail.com>
wrote:
> > >
> > > > Jonathan:
> > > > Since you implemented
> > > >
> > > >
> > >
> >
>
https://issues.apache.org/jira/si/jira.issueviews:issue-html/HBASE-48/HB
> > > ASE-48.html
> > > > ,
> > > > maybe you can point me to some document how bulk load is used ?
> > > > I found bin/loadtable.rb and assume that can be used to import
> data
> > > back
> > > > into HBase.
> > > >
> > > > Thanks
> > > >
> > >
> > > This email is sent for and on behalf of Ivy Comptech Private
> Limited.
> > Ivy
> > > Comptech Private Limited is a limited liability company.
> > >
> > > This email and any attachments are confidential, and may be
legally
> > > privileged and protected by copyright. If you are not the intended
> > recipient
> > > dissemination or copying of this email is prohibited. If you have
> > received
> > > this in error, please notify the sender by replying by email and
> then
> > delete
> > > the email completely from your system.
> > > Any views or opinions are solely those of the sender.  This
> > communication
> > > is not intended to form a binding contract on behalf of Ivy
Comptech
> > Private
> > > Limited unless expressly indicated to the contrary and properly
> > authorised.
> > > Any actions taken on the basis of this email are at the
recipient's
> > own
> > > risk.
> > >
> > > Registered office:
> > > Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara
> Hills,
> > > Hyderabad 500 033, Andhra Pradesh, India. Registered number:
37994.
> > > Registered in India. A list of members' names is available for
> > inspection at
> > > the registered office.
> > >
> > >
> >
>

Mime
View raw message