hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niels Basjes <Ni...@basjes.nl>
Subject Re: stop generating these "part-XXXX" empty files when using MultipleOutputs in mapreduce job
Date Mon, 28 Oct 2013 19:31:55 GMT
Use the LazyOutputFormat.

Have a look at this:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/output/LazyOutputFormat.html
and
http://stackoverflow.com/questions/6137139/how-to-save-only-non-empty-reducers-output-in-hdfs

Niels Basjes


On Mon, Oct 28, 2013 at 8:11 PM, S. Zhou <myxjtu@yahoo.com> wrote:

> I use MultipleOutputs so the output data are no longer stored in files
> "part-XXX". But they are still generated (though empty). Is it possible to
> stop generating these files when running MR job? (BTW, my MR job only has
> mapper). Thanks
>
> Senqiang
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Mime
View raw message