hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Meng Mao <meng...@gmail.com>
Subject Re: Is it possible to write each key-value pair emitted by the reducer to a different output file
Date Sat, 06 Feb 2010 23:49:18 GMT
It's possible to write your own class to take better encapsulate writing of
side-effect files, but as people have said, you can run into unanticipated
issues if the number of files you try to write at once becomes high.

On Sat, Feb 6, 2010 at 3:47 AM, Udaya Lakshmi <udaya603@gmail.com> wrote:

> Hi Amareshwari,
>
>   But this feature is not available in Hadoop 0.18.3. Is there any work
> around for this version.
> Thanks,
> Udaya.
>
> On Fri, Feb 5, 2010 at 10:49 AM, Amareshwari Sri Ramadasu <
> amarsri@yahoo-inc.com> wrote:
>
> > See MultipleOutputs at
> >
> http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
> >
> > -Amareshwari
> >
> > On 2/5/10 10:41 AM, "Udaya Lakshmi" <udaya603@gmail.com> wrote:
> >
> > Hi,
> >  I was wondering if it is possible to write each key-value pair produced
> by
> > the reduce function to a different file. How could I open a new file in
> the
> > reduce function of the reducer? I know its possible in configure function
> > but it will write all the output that reducer to that file.
> > Thanks,
> > Udaya.
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message