incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eran Kutner <e...@gigya.com>
Subject Can't get LZO codec to work
Date Tue, 02 Aug 2011 09:16:09 GMT
Hi,
I'm trying to enable the LZO codec in flume.
I'm using flume 0.9.4 from CDH3
I placed the hadopp-lzo.jar file in flume's lib directory.
I added the native directory to the java lib path in flume-env.sh
I verified, using "flume classpath" that the codec jar file is indeed
included in the flume classpath.

This command is working fine:
java -classpath "/usr/lib/flume/lib/*"
-Djava.library.path=/usr/lib/hadoop/lib/native/Linux-amd64-64
com.hadoop.compression.lzo.LzoIndexer /tmp/test.lzo


However, when I try to enable LzopCodec I get this error:
11/08/02 04:45:35 INFO connector.DirectDriver: Connector logicalNode
collector1-44 exited with error: Unsupported compression codec LzopCodec.
Please choose from: [None, DefaultCodec, GzipCodec, BZip2Codec,
DeflateCodec, SnappyCodec]
java.lang.IllegalArgumentException: Unsupported compression codec
LzopCodec.  Please choose from: [None, DefaultCodec, GzipCodec, BZip2Codec,
DeflateCodec, SnappyCodec]
        at
com.cloudera.flume.handlers.hdfs.CustomDfsSink.getCodec(CustomDfsSink.java:180)
        at
com.cloudera.flume.handlers.hdfs.CustomDfsSink.open(CustomDfsSink.java:109)
        at
com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.openWriter(EscapedCustomDfsSink.java:104)
        at
com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.append(EscapedCustomDfsSink.java:119)
        at
com.cloudera.flume.core.CompositeSink.append(CompositeSink.java:61)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.handlers.rolling.RollSink.synchronousAppend(RollSink.java:234)
        at
com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:183)
        at
com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:181)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)


Note that it doesn't even list the LzoCodec in available codecs list. Any
idea why?

-eran

Mime
View raw message