flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eran Kutner <e...@gigya.com>
Subject Can't use snappy codec
Date Fri, 03 Aug 2012 14:33:25 GMT
Hi,
I'm trying to use the snappy codec but keep getting "native snappy library
not available" errors.
I'm using CDH4 but replaced the flume 1.1 JARs that are included with that
distribution with flume 1.2 JARs.
I tried anything I can think of, including symlinking the hadoop native
library under flume-ng/lib/ dirctory both nothing helps.
Any idea how to resolve this?

This is the error:
2012-08-03 10:23:30,598 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2012-08-03 10:23:35,670 WARN hdfs.HDFSEventSink: HDFS IO error
java.io.IOException: java.lang.RuntimeException: native snappy library not
available
        at
org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:202)
        at
org.apache.flume.sink.hdfs.BucketWriter.access$000(BucketWriter.java:48)
        at
org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:155)
        at
org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:152)
        at
org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:125)
        at
org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:152)
        at
org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:307)
        at
org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:717)
        at
org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:714)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: native snappy library not available
        at
org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:135)
        at
org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:84)
        at
org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:70)
        at
org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:195)
        ... 13 more

And my sink configuration:
flume05.sinks.hdfsSink.type = hdfs
#flume05.sinks.hdfsSink.type = logger
flume05.sinks.hdfsSink.channel = memoryChannel
flume05.sinks.hdfsSink.hdfs.path=hdfs://hadoop2-m1:8020/test-events/%Y-%m-%d
flume05.sinks.hdfsSink.hdfs.filePrefix=raw-events.avro
flume05.sinks.hdfsSink.hdfs.rollInterval=60
flume05.sinks.hdfsSink.hdfs.rollCount=0
flume05.sinks.hdfsSink.hdfs.rollSize=0
flume05.sinks.hdfsSink.hdfs.fileType=CompressedStream
flume05.sinks.hdfsSink.hdfs.codeC=snappy
flume05.sinks.hdfsSink.hdfs.writeFormat=Text
flume05.sinks.hdfsSink.hdfs.batchSize=1000
flume05.sinks.hdfsSink.serializer = avro_event

Thanks.

-eran

Mime
View raw message