Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Doug Meil <doug.meil@explorysmedical.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Date: Tue, 31 May 2011 10:22:34 -0400
Subject: RE: How to efficiently join HBase tables?
Thread-Topic: How to efficiently join HBase tables?
Thread-Index: Acwfiwk7nB6iTmkxRrG+6myO5ut1vwAEpsPg
Message-ID: <67680900F79B1D4F99C844EE386FC5952823B100A5@EX2K7VS03.4emm.local>
References: <BANLkTim+SrXvh5yxpQjrTRb=AnAgwC8B9Q@mail.gmail.com>
In-Reply-To: <BANLkTim+SrXvh5yxpQjrTRb=AnAgwC8B9Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


Re:  "The problem is that the few references to that question I found recom=
mend pulling one table to the mapper and then do a lookup for the referred =
row in the second table."

With multi-get in .90.x you could perform some reasonably clever processing=
 and not do the lookups one-by-one but in batches.

Also, if the other table is "small" you could have the leverage the block c=
ache on the lookups (i.e., if it's a domain/lookup table). =20


-----Original Message-----
From: eran@gigya-inc.com [mailto:eran@gigya-inc.com] On Behalf Of Eran Kutn=
er
Sent: Tuesday, May 31, 2011 8:06 AM
To: user@hbase.apache.org
Subject: How to efficiently join HBase tables?

Hi,
I need to join two HBase tables. The obvious way is to use a M/R job for th=
at. The problem is that the few references to that question I found recomme=
nd pulling one table to the mapper and then do a lookup for the referred ro=
w in the second table.
This sounds like a very inefficient way to do  join with map reduce. I beli=
eve it would be much better to feed the rows of both tables to the mapper a=
nd let it emit a key based on the join fields. Since all the rows with the =
same join fields values will have the same key the reducer will be able to =
easily generate the result of the join.
The problem with this is that I couldn't find a way to feed two tables to a=
 single map reduce job. I could probably dump the tables to files in a sing=
le directory and then run the join on the files but that really makes no se=
nse.

Am I missing something? Any other ideas?

-eran