hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bejoy Ks <bejoy.had...@gmail.com>
Subject Re: reduce output compression of Terasort
Date Fri, 17 Feb 2012 09:18:41 GMT
Hi Juwei
       What is the value for mapred.output.compression.codec? It'd be
better to determine whether the output files are compressed by getting the
codec of the same and not just from the size of files.

Regards
Bejoy.K.S

On Fri, Feb 17, 2012 at 12:07 PM, Juwei Shi <shijuwei@gmail.com> wrote:

> Hi,
>
> I am benchmarking the cluster using the Terasort package of Hadoop 0.20.2.
> I enabled compression for both map output (*mapred.compress.map.output*)
> and reduce output (*mapred.output.compress*). I checked the parameter in
> Job.xml, both are true. I can see that the compression for Map output
> works, but it seems that the compression for reduce output does not work.
> The output of the job on HDFS is also 1TB.
>
> Thanks!
>
> - Juwei
>

Mime
View raw message