hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <j...@cloudera.com>
Subject Re: Map-only output compression
Date Mon, 07 Nov 2011 14:57:01 GMT
You want option 3.

Option 1 is only used to compress intermediate output, it doesn't apply to
map only jobs.
Option 2 only enables compression for SequenceFileOutputFormat. If you're
not using that output format, it won't help.

-Joey

On Monday, November 7, 2011, Claudio Martella wrote:

> Hello list,
>
> I have a map-only job and I'd like to compress the output (possibly
> avoiding a re-compression when the map-output gets promoted as final
> output).
> I can see 4 ways of obtaining it:
>
> 1) by defining map to compress through mapred.compress.map.output.*
> 2) by defining output to compress through mapred.output.compression.*
> 3) by defining the TextOutputFormat to compress through
> TextOutputFormat.setCompressOutput()
> 4) by composing one or more of the first 3 possibilities
>
> Any insight about how to do this properly? I'm running hadoop 0.20.204.0
>
>
> --
> Claudio Martella
> Free Software & Open Technologies
> Analyst
>
> TIS innovation park
> Via Siemens 19 | Siemensstr. 19
> 39100 Bolzano | 39100 Bozen
> Tel. +39 0471 068 123
> Fax  +39 0471 068 129
> claudio.martella@tis.bz.it <javascript:;> http://www.tis.bz.it
>
> Short information regarding use of personal data. According to Section 13
> of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that
> we process your personal data in order to fulfil contractual and fiscal
> obligations and also to send you information regarding our services and
> events. Your personal data are processed with and without electronic means
> and by respecting data subjects' rights, fundamental freedoms and dignity,
> particularly with regard to confidentiality, personal identity and the
> right to personal data protection. At any time and without formalities you
> can write an e-mail to privacy@tis.bz.it <javascript:;> in order to
> object the processing of your personal data for the purpose of sending
> advertising materials and also to exercise the right to access personal
> data and other rights referred to in Section 7 of Decree 196/2003. The data
> controller is TIS Techno Innovation Alto Adige, Siemens Street n. 19,
> Bolzano. You can find the complete information on the web site
> www.tis.bz.it.
>
>
>
>
>

-- 
Joseph Echeverria
Cloudera, Inc.
443.305.9434

Mime
View raw message