hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: job.setOutputFormatClass(NullOutputFormat.class);
Date Wed, 02 Jul 2014 02:19:15 GMT
To get rid of empty *part files while using MultipleOutputs in the new API,
LazyOutputFormat class' static method should be used to set the output
format.

Details are here at the official Java docs for MultipleOutputs :
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html

Regards,
Shahab


On Tue, Jul 1, 2014 at 9:08 PM, M. Dale <medale94@yahoo.com> wrote:

>  NullOutputFormat does not generate any output. Good for jobs where
> counters or some other I/O are your output (for example,
> http://stackoverflow.com/questions/12707726/run-a-hadoop-job-without-output-file
> ).
>
> From Tom White's book it sounds like
> org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat only generates a
> file if at least one key/value pair is written.
>
> Hope this helps,
> Markus
>
>
> On 07/01/2014 07:33 AM, Chris MacKenzie wrote:
>
> Hi,
>
> What is the anticipated usage of the above with the new api ? Is there
> another way to remove the empty part-r files
>
> When using it with MultipleOutputs to remove empty part-r files I have no
> output ;O)
>
>
>
> Regards,
>
> Chris MacKenzie<http://www.chrismackenziephotography.co.uk/> <http://www.chrismackenziephotography.co.uk/>
>
>
>

Mime
View raw message