hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "fang fang chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11418) Property "io.compression.codec.lzo.class" does not work with other value besides default
Date Tue, 03 Mar 2015 10:04:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344854#comment-14344854
] 

fang fang chen commented on HADOOP-11418:
-----------------------------------------

Sure, thanks for looking at this JIRA, Akira. I generated new patch based on current trunk.
And the regression test has been made with pig before, here is the process and log:
1. Add following settings in hadoop core-site.xml:
<property>
<name>io.compression.codecs</name>
<value>com.ibm.biginsights.compress.CmxCodec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.ibm.biginsights.compress.CmxCodec</value>
</property>
2. Launch pig grunt with properties:
-Dpig.tmpfilecompression=true -Dpig.tmpfilecompression.codec=cmx -Dpig.additional.jars=ibm-compression.jar:libcmxbiginsights.so

Together with patch in PIG-4341, test case passed with following massage:
2014-12-22 17:37:24,453 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
- Total input paths to process : 2
2014-12-22 17:37:24,550 [main] INFO org.apache.hadoop.io.file.tfile.Compression - Trying to
load Lzo codec class: com.ibm.biginsights.compress.CmxCodec
2014-12-22 17:37:24,554 [main] INFO com.ibm.biginsights.compress.NativeCodeLoader - Loaded
the native library
2014-12-22 17:37:24,554 [main] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new
decompressor [.cmx]

> Property "io.compression.codec.lzo.class" does not work with other value besides default
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-11418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11418
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: fang fang chen
>            Assignee: fang fang chen
>         Attachments: HADOOP-11418-1.patch, HADOOP-11418.patch
>
>
> From following code, seems "io.compression.codec.lzo.class" does not work for other codec
besides default. Hadoop will always treat it as defaultClazz. I think it is a bug. Please
let me know if this is a work as design thing. Thanks
>      77       private static final String defaultClazz =
>      78           "org.apache.hadoop.io.compress.LzoCodec";
>      82       public synchronized boolean isSupported() {
>      83         if (!checked) {
>      84           checked = true;
>      85           String extClazz =
>      86               (conf.get(CONF_LZO_CLASS) == null ? System
>      87                   .getProperty(CONF_LZO_CLASS) : null);
>      88           String clazz = (extClazz != null) ? extClazz : defaultClazz;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message