hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lulynn_2008 <lulynn_2...@163.com>
Subject HADOOP-11418:Property "io.compression.codec.lzo.class" does not work with other value besides default
Date Wed, 17 Dec 2014 11:07:13 GMT
Hi All,

Could I try other codec besides lzo using hadoop-common TFile property "io.compression.codec.lzo.class"?
If not, please ignore following information.

I was trying "io.compression.codec.lzo.class" with value "com.ibm.biginsights.compress.CmxCodec"
(not the default "org.apache.hadoop.io.compress.LzoCodec"). The jar for this CmxCodec is called
ibm-compression.jar. But I always encountered following errors:
"java.io.IOException: LZO codec class not specified. Did you forget to set property io.compression.codec.lzo.class?"

I lunched mapreduce jobs and from "job*.xml" I could find this property was set correctly.
And also could find ibm-compression.jar in "java.class.path".
From following code, seems "io.compression.codec.lzo.class" does not work for other codec
besides default. Hadoop will always treat it as defaultClazz. I think it is a bug and opened
this jira. Please help give a glance and give your comments. Thanks

     77       private static final String defaultClazz =
     78           "org.apache.hadoop.io.compress.LzoCodec";

     82       public synchronized boolean isSupported() {
     83         if (!checked) {
     84           checked = true;
     85           String extClazz =
     86               (conf.get(CONF_LZO_CLASS) == null ? System
     87                   .getProperty(CONF_LZO_CLASS) : null);
     88           String clazz = (extClazz != null) ? extClazz : defaultClazz;

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message