hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject Re: MultipleOutputFormat
Date Wed, 30 Mar 2011 04:09:49 GMT
You should be able to create partitions on the fly.

Check the last example in the javadocs:

http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html

 ...

  mos.write(key, new Text("value"), generateFileName(key, new Text("value")));

Hope this helps.

Alejandro

On Wed, Mar 30, 2011 at 12:02 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>wrote:

> yes.. but in my old code the file names are created on the fly (it
> basically creates partitions based on a time field). I dont think
> MultipleOutputs is not suitable to create partitions on the fly.
>
> On Tue, Mar 29, 2011 at 8:56 PM, Alejandro Abdelnur <tucu@cloudera.com>
> wrote:
> > Dmitriy,
> > Have you check the MultipleOutputs instead? It provides similar
> > functionality.
> > Alejandro
> >
> > On Wed, Mar 30, 2011 at 11:39 AM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> > wrote:
> >>
> >> Hi,
> >> I can't seem to be able to find either jira or implementation of
> >> MultipleOutputFormat in new api in either 0.21 or 0.22 branches.
> >> Are there any plans to port that to new api as well?
> >>
> >> thanks in advance.
> >> -Dmitriy
> >
> >
>

Mime
View raw message