hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankil Doshi <forpan...@gmail.com>
Subject Re: Sort-Merge Join using Map Reduce.
Date Thu, 09 Jul 2009 18:12:30 GMT
Dear Todd,

I got the concept but I have no idea about side input in mapper class.
Can you guide me more on that?

Pankil

On Thu, Jul 9, 2009 at 1:39 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Pankil,
>
> Basically there are two steps here - the first is to sort the two files.
> This can be done using an mapreduce where the mapper extracts the join
> column as a key.
>
> If you make sure you have the same number of reducers (and partition by the
> equijoin column) for both sorts, then you'll end up with:
>
> A        B
> part-0  part-0
> part-1  part-1
>
> etc
>
> Each corresponding part file will be in sorted order, and you can perform
> the merge.
>
> To do the merge, you can just pick either A or B as your input for locality
> hints, and then, in the mapper, given the file name, determine the filename
> of the other partition. Open that up as a side input in your mapper and
> perform the merge like you would in a non-distributed setting.
>
> Hope this helps
> -Todd
>
>
> On Thu, Jul 9, 2009 at 9:09 AM, Pankil Doshi <forpankil@gmail.com> wrote:
>
> > Hi,
> >
> > Does anyone has hint on how to implement "SORT-MERGE JOIN" using
> map-reduce
> > paradigm?
> > I read article regarding it on Pig wiki but did not got clarity as it
> > doesn't show in form of map and reduce.
> >
> > Pankil
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message