flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eran Kutner <e...@gigya.com>
Subject Re: Can't use snappy codec
Date Fri, 03 Aug 2012 16:25:35 GMT
Thanks Patrick, that helped me figure out the problem and it looks like a
bug in the "flume-ng" file provided with CDH4, it was messing the
library.path.
I copied the file that was included in flume 1.2.0 distribution and it now
works ok.

Thanks for your help.

-eran



On Fri, Aug 3, 2012 at 6:36 PM, Patrick Wendell <pwendell@gmail.com> wrote:

> Hey Eran,
>
> You need to make sure the Flume JVM gets passed
> -Djava.library.path=XXX with the correct path to where your native
> snappy libraries are located.
>
> You can set this by adding the option directly to the flume-ng runner
> script.
>
> - Patrick
>
> On Fri, Aug 3, 2012 at 7:33 AM, Eran Kutner <eran@gigya.com> wrote:
> > Hi,
> > I'm trying to use the snappy codec but keep getting "native snappy
> library
> > not available" errors.
> > I'm using CDH4 but replaced the flume 1.1 JARs that are included with
> that
> > distribution with flume 1.2 JARs.
> > I tried anything I can think of, including symlinking the hadoop native
> > library under flume-ng/lib/ dirctory both nothing helps.
> > Any idea how to resolve this?
> >
> > This is the error:
> > 2012-08-03 10:23:30,598 WARN util.NativeCodeLoader: Unable to load
> > native-hadoop library for your platform... using builtin-java classes
> where
> > applicable
> > 2012-08-03 10:23:35,670 WARN hdfs.HDFSEventSink: HDFS IO error
> > java.io.IOException: java.lang.RuntimeException: native snappy library
> not
> > available
> >         at
> > org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:202)
> >         at
> > org.apache.flume.sink.hdfs.BucketWriter.access$000(BucketWriter.java:48)
> >         at
> > org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:155)
> >         at
> > org.apache.flume.sink.hdfs.BucketWriter$1.run(BucketWriter.java:152)
> >         at
> >
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:125)
> >         at
> > org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:152)
> >         at
> > org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:307)
> >         at
> > org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:717)
> >         at
> > org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:714)
> >         at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >         at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.lang.RuntimeException: native snappy library not
> available
> >         at
> >
> org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:135)
> >         at
> >
> org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:84)
> >         at
> >
> org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:70)
> >         at
> > org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:195)
> >         ... 13 more
> >
> > And my sink configuration:
> > flume05.sinks.hdfsSink.type = hdfs
> > #flume05.sinks.hdfsSink.type = logger
> > flume05.sinks.hdfsSink.channel = memoryChannel
> >
> flume05.sinks.hdfsSink.hdfs.path=hdfs://hadoop2-m1:8020/test-events/%Y-%m-%d
> > flume05.sinks.hdfsSink.hdfs.filePrefix=raw-events.avro
> > flume05.sinks.hdfsSink.hdfs.rollInterval=60
> > flume05.sinks.hdfsSink.hdfs.rollCount=0
> > flume05.sinks.hdfsSink.hdfs.rollSize=0
> > flume05.sinks.hdfsSink.hdfs.fileType=CompressedStream
> > flume05.sinks.hdfsSink.hdfs.codeC=snappy
> > flume05.sinks.hdfsSink.hdfs.writeFormat=Text
> > flume05.sinks.hdfsSink.hdfs.batchSize=1000
> > flume05.sinks.hdfsSink.serializer = avro_event
> >
> > Thanks.
> >
> > -eran
> >
>

Mime
View raw message