incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Can't get LZO codec to work
Date Tue, 02 Aug 2011 13:12:30 GMT
Does your HADOOP_CONF_DIR/core-site.xml have its io.compression.codecs
property configured to include LZO? Can you ensure your Flume picks up
the same configuration? You could also try to make it <final> to
enforce it and prevent overriding.

On Tue, Aug 2, 2011 at 2:46 PM, Eran Kutner <eran@gigya.com> wrote:
> Hi,
> I'm trying to enable the LZO codec in flume.
> I'm using flume 0.9.4 from CDH3
> I placed the hadopp-lzo.jar file in flume's lib directory.
> I added the native directory to the java lib path in flume-env.sh
> I verified, using "flume classpath" that the codec jar file is indeed
> included in the flume classpath.
>
> This command is working fine:
> java -classpath "/usr/lib/flume/lib/*"
> -Djava.library.path=/usr/lib/hadoop/lib/native/Linux-amd64-64
> com.hadoop.compression.lzo.LzoIndexer /tmp/test.lzo
>
>
> However, when I try to enable LzopCodec I get this error:
> 11/08/02 04:45:35 INFO connector.DirectDriver: Connector logicalNode
> collector1-44 exited with error: Unsupported compression codec LzopCodec.
> Please choose from: [None, DefaultCodec, GzipCodec, BZip2Codec,
> DeflateCodec, SnappyCodec]
> java.lang.IllegalArgumentException: Unsupported compression codec
> LzopCodec.  Please choose from: [None, DefaultCodec, GzipCodec, BZip2Codec,
> DeflateCodec, SnappyCodec]
>         at
> com.cloudera.flume.handlers.hdfs.CustomDfsSink.getCodec(CustomDfsSink.java:180)
>         at
> com.cloudera.flume.handlers.hdfs.CustomDfsSink.open(CustomDfsSink.java:109)
>         at
> com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.openWriter(EscapedCustomDfsSink.java:104)
>         at
> com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.append(EscapedCustomDfsSink.java:119)
>         at
> com.cloudera.flume.core.CompositeSink.append(CompositeSink.java:61)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.rolling.RollSink.synchronousAppend(RollSink.java:234)
>         at
> com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:183)
>         at
> com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:181)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
>
> Note that it doesn't even list the LzoCodec in available codecs list. Any
> idea why?
>
> -eran
>
>



-- 
Harsh J

Mime
View raw message