flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: native snappy library not available
Date Wed, 19 Oct 2016 12:46:47 GMT
Hi Robert,

have you tried putting the snappy java jar in Flink's lib folder? When
specifying the classpath manually you have to make sure that all
distributed components are also started with this classpath.

Cheers,
Till

On Wed, Oct 19, 2016 at 1:07 PM, <robert.lancaster@hyatt.com> wrote:

> I have flink running on a standalone cluster, but with a Hadoop cluster
> available to it.  I’m using a RollingSink and a SequenceFileWriter, and
> I’ve set compression to Snappy:
>
>
>
> .setWriter(new SequenceFileWriter<Text, Text>("org.apache.hadoop.io.compress.SnappyCodec",
> SequenceFile.CompressionType.*BLOCK*))
>
>
>
>
>
> However, when I try to run this job, I get an error indicating that the
> native snappy ibrary is not available:
>
>
>
> Caused by: java.lang.RuntimeException: native snappy library not
> available: this version of libhadoop was built without snappy support.
>
>         at org.apache.hadoop.io.compress.SnappyCodec.
> checkNativeCodeLoaded(SnappyCodec.java:65)
>
>
>
> I’ve tried adding the snappy java jar to the classpath.  I’ve also tried
> adding the native dir to the classpath.  Neither have helped.
>
> -C file://usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar
>
> -C file://usr/hdp/current/hadoop/lib/native/
>
>
>
> Any thoughts on how I can get this to work?
>
>
>
> Thanks!
>
>
>
> ------------------------------
> The information contained in this communication is confidential and
> intended only for the use of the recipient named above, and may be legally
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any dissemination, distribution or copying of this communication is
> strictly prohibited. If you have received this communication in error,
> please resend it to the sender and delete the original message and copy of
> it from your computer system. Opinions, conclusions and other information
> in this message that do not relate to our official business should be
> understood as neither given nor endorsed by the company.
>

Mime
View raw message