hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Map output compression in Hadoop 1.0.3
Date Wed, 28 Nov 2012 11:25:00 GMT

The property mapred.output.compress, as its name reads, controls
job-output compression, not intermediate/transient data compression,
which is what you mean by "Map output compression".

Also note that this property is a per job one and can be toggled, if a
user wanted, on/off for each job specifically.

These should be the many ways, exhaustively, for MR1, to turn on "Map
output compression":

1. Set "mapred.compress.map.output" to true in your client's
mapred-site.xml to turn it on for all jobs run from such a client
2. Set the above in cluster, with <final>true</final> at every node
(JT plus TTs) and restart them, to turn it on for all job, regardless
of what the job itself specifies.
3. Turn it on per-job basis:
3.1. Stable API: JobConf.setCompressMapOutput(true);
3.2. New API: Job.getConfiguration().set("mapred.compress.map.output", true);

On Wed, Nov 28, 2012 at 4:42 PM, Tony Burton <TBurton@sportingindex.com> wrote:
> Hi,
> Quick question: What’s the best way to turn on Map Output Compression in
> Hadoop 1.0.3? The tutorial at
> http://hadoop.apache.org/docs/r1.0.3/mapred_tutorial.html says to use
> JobConf.setCompressMapOutput(boolean), but I’m using o.a.h.mapreduce.Job
> rather than o.a.h.mapred.JobConf.
> Is it simply a case of using getConf.set("mapred.output.compress", true)
> then constructing my Job from the Configuration object, or is there more
> direct way that I’ve missed?
> Thanks,
> Tony
> *****************************************************************************
> P Please consider the environment before printing this email or attachments
> This email and any attachments are confidential, protected by copyright and
> may be legally privileged. If you are not the intended recipient, then the
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system. Neither Sporting Index nor the sender
> accepts responsibility for any virus, or any other defect which might affect
> any computer or IT system into which the email is received and/or opened. It
> is the responsibility of the recipient to scan the email and no
> responsibility is accepted for any loss or damage arising in any way from
> receipt or use of this email. Sporting Index Ltd is a company registered in
> England and Wales with company number 2636842, whose registered office is at
> Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is
> authorised and regulated by the UK Financial Services Authority (reg. no.
> 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any
> financial promotion contained herein has been issued and approved by
> Sporting Index Ltd.
> Outbound email has been scanned for viruses and SPAM

Harsh J

View raw message