hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1194) map output should not do block level compression
Date Tue, 03 Apr 2007 03:17:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486213

Arun C Murthy commented on HADOOP-1194:


'mapred.compress.map.output' is the boolean which controls whether map outputs are compressed.
If set to true (@see MapTask.java:279) the 'mapred.output.compression.type' can be used to
specify RECORD/BLOCK - this defaults to SequenceFile's compression-type i.e. 'io.seqfile.compression.type'
which defaults to RECORD compression. (@see JobConf.getMapOutputCompressionType).

Does this help? Or do you propose we change JobConf.getMapOutputCompressionType to return
RECORD if 'mapred.output.compression.type' is unset i.e. do not use io.seqfile.compression.type?
Thanks for the clarification...

> map output should not do block level compression
> ------------------------------------------------
>                 Key: HADOOP-1194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1194
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.12.2
>            Reporter: Runping Qi
> If the user sets to compress the map output, the compression style should be record level,
not block level, since using block level compression for map outputs causes performance degragation

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message