hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ma qiang" <maqiang1...@gmail.com>
Subject Re: how to use two reduce fucntions?
Date Mon, 25 Feb 2008 04:31:27 GMT
Thanks for your reply. I meet this problem as below: I have a
application that need to use two reduce phase. In my first reduce
function, I divided all the data into several keys which will be use
in the second reduce function, in addition,in my second reduce
function, it will computer some values using data from the result of
the first reduce function . The result of the first reduce function is
the input data of the second reduce function.
Or I run two jobs, but in this case the map function of the second job
will do nothing except some IO .

On Sun, Feb 24, 2008 at 3:29 AM, Jason Venner <jason@attributor.com> wrote:
> If you set up a partitioner class, you could pre partition the output of
>  the into the relevant segments.
>  Then your reducer would be responsible for determining which reduce
>  function to apply based on which segment the key is part of.
>  Amar Kamat wrote:
>  > Can you provide more details on what exactly what you wish to do? What
>  > is the nature of reducers? A simple answer would be with map(m) and
>  > reducers(r1,r2) you can run 2 jobs i.e job1(m,r1) and
>  > job2(IdentityMapper,r2). But it depends what exactly r1 and r2 do.
>  > Also combiners will play an important role. Also can one merge r1 and
>  > r2 to r and run a job(m,r)
>  > Amar
>  > On Sat, 23 Feb 2008, ma qiang wrote:
>  >
>  >> Hi all,
>  >>    I have a program need to use two reduce fucntions, who can tell me
>  >> why?
>  >>    Thank you!
>  >>
>  >> Qiang
>  >>
>  --
>  Jason Venner
>  Attributor - Publish with Confidence <http://www.attributor.com/>
>  Attributor is hiring Hadoop Wranglers, contact if interested

View raw message