hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Graham <billgra...@gmail.com>
Subject Re: How to efficiently join HBase tables?
Date Wed, 01 Jun 2011 00:35:25 GMT
We use Pig to join HBase tables using HBaseStorage which has worked well. If
you're using HBase >= 0.89 you'll need to build from the trunk or the Pig
0.8 branch.


On Tue, May 31, 2011 at 5:18 PM, Jason Rutherglen <
jason.rutherglen@gmail.com> wrote:

> > The Hive-HBase integration allows you to create Hive tables that are
> backed
> > by HBase
>
> In addition, HBase can be made to go faster for MapReduce jobs, if the
> HFile's could be used directly in HDFS, rather than proxying through
> the RegionServer.
>
> I'd imagine that join operations do not require realtime-ness, and so
> faster batch jobs using Hive -> frozen HBase files in HDFS could be
> the optimal way to go?
>
> On Tue, May 31, 2011 at 1:41 PM, Patrick Angeles <patrick@cloudera.com>
> wrote:
> > On Tue, May 31, 2011 at 3:19 PM, Eran Kutner <eran@gigya.com> wrote:
> >
> >> For my need I don't really need the general case, but even if I did I
> think
> >> it can probably be done simpler.
> >> The main problem is getting the data from both tables into the same MR
> job,
> >> without resorting to lookups. So without the theoretical
> >> MutliTableInputFormat, I could just copy all the data from both tables
> into
> >> a temp table, just append the source table name to the row keys to make
> >> sure
> >> there are no conflicts. When all the data from both tables is in the
> same
> >> temp table, run a MR job. For each row the mapper should emit a key
> which
> >> is
> >> composed of all the values of the join fields in that row (the value can
> be
> >> emitted as is). This will cause all the rows from both tables, with same
> >> join field values to arrive at the reducer together. The reducer could
> then
> >> iterate over them and produce the Cartesian product as needed.
> >>
> >> I still don't like having to copy all the data into a temp table just
> >> because I can't feed two tables into the MR job.
> >>
> >
> > Loading the smaller table in memory is called a map join, versus a
> > reduce-side join (a.k.a. common join). One reason to prefer a map join is
> > you avoid the shuffle phase which potentially involves several trips to
> disk
> > for the intermediate records due to spills, and also once through the
> > network to get each intermediate KV pair to the right reducer. With a map
> > join, everything is local, except for the part where you load the small
> > table.
> >
> >
> >>
> >> As Jason Rutherglen mentioned above, Hive can do joins. I don't know if
> it
> >> can do them for HBase and it will not suit my needs, but it would be
> >> interesting to know how is it doing them, if anyone knows.
> >>
> >
> > The Hive-HBase integration allows you to create Hive tables that are
> backed
> > by HBase. You can do joins on those tables (and also with standard Hive
> > tables). It might be worth trying out in your case as it lets you easily
> see
> > the load characteristics and the job runtime without much coding
> investment.
> >
> > There are probably some specific optimizations that can be applied to
> your
> > situation, but it's hard to say without knowing your use-case.
> >
> > Regards,
> >
> > - Patrick
> >
> >
> >> -eran
> >>
> >>
> >>
> >> On Tue, May 31, 2011 at 22:02, Ted Dunning <tdunning@maprtech.com>
> wrote:
> >>
> >> > The Cartesian product often makes an honest-to-god join not such a
> good
> >> > idea
> >> > on large data.  The common alternative is co-group
> >> > which is basically like doing the hard work of the join, but involves
> >> > stopping just before emitting the cartesian product.  This allows
> >> > you to inject whatever cleverness you need at this point.
> >> >
> >> > Common kinds of cleverness include down-sampling of problematically
> large
> >> > sets of candidates.
> >> >
> >> > On Tue, May 31, 2011 at 11:56 AM, Michael Segel
> >> > <michael_segel@hotmail.com>wrote:
> >> >
> >> > > So the underlying problem that the OP was trying to solve was how
to
> >> join
> >> > > two tables from HBase.
> >> > > Unfortunately I goofed.
> >> > > I gave a quick and dirty solution that is a bit incomplete. They row
> >> key
> >> > in
> >> > > the temp table has to be unique and I forgot about the Cartesian
> >> > > product. So my solution wouldn't work in the general case.
> >> > >
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message