hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramy Bhats <pramybh...@googlemail.com>
Subject Re: Intermediate files generated.
Date Fri, 02 Jul 2010 06:05:25 GMT
Hi Hemanth,

I need to use the output of the mapper for some other application. As a
result, if I can redirect the output of the map in temp files of my choice
(which are stored on hdfs) then i can reuse the output later. At the same
time, the succeeding reducer can read the input from this temp files without
any overhead.

thanks,
--PB

On Fri, Jul 2, 2010 at 3:52 AM, Hemanth Yamijala <yhemanth@gmail.com> wrote:

> Alex,
>
> > I don't think this is what I am looking for. Essential, I wish to run
> both
> > mapper as well as reducer. But at the same time, i wish to make sure that
> > the temp files that are used between mappers and reducers are of my
> choice.
> > Here, the choice means that I can specify the files in HDFS that can be
> used
> > as temp files.
>
> Could you explain why you want to do this ?
>
> >
> > thanks,
> > --PB.
> >
> > On Fri, Jul 2, 2010 at 12:14 AM, Alex Loddengaard <alex@cloudera.com>
> wrote:
> >
> >> You could use the HDFS API from within your mapper, and run with 0
> >> reducers.
> >>
> >> Alex
> >>
> >> On Thu, Jul 1, 2010 at 3:07 PM, Pramy Bhats <pramybhats@googlemail.com
> >> >wrote:
> >>
> >> > Hi,
> >> >
> >> > I am using hadoop framework for writing MapReduce jobs. I want  to
> >> redirect
> >> > the output of Map into files of my choice and later use those files as
> >> > input
> >> > for Reduce phase.
> >> >
> >> >
> >> > Could you please suggest, how to proceed for it ?
> >> >
> >> > thanks,
> >> > --PB.
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message