hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Azuryy Yu <azury...@gmail.com>
Subject Re: Snappy Configuration in Hadoop2.5.2
Date Mon, 16 Mar 2015 06:43:44 GMT
Hi,
please :
run "cp -a /opt/snappy/lib/libsnappy.* /opt/hadoop/hadoophome/lib/native"
on each datanode. also you need to install Snappy on each datanode firstly.







On Sat, Mar 7, 2015 at 6:57 PM, donhoff_h <165612158@qq.com> wrote:

> Hi, experts.
>
> I meet the following problem when configuring the Snappy lib in Hadoop2.5.2
>
> My snappy installation home is /opt/snappy
> My Hadoop installation home is /opt/hadoop/hadoophome
>
> To configure the snappy path, I tried to add the following environment
> variables in /etc/profile and hadoop-env.sh :
> export JAVA_LIBRARY_PATH=/opt/hadoop/hadoophome/lib/native:/opt/snappy/lib
> export LD_LIBRARY_PATH=/opt/hadoop/hadoophome/lib/native:/opt/snappy/lib‍
>
> After the configuration, I ran the command "hadoop checknative". The
> result showed as following which I think means the hadoop can find the
> snappy lib:
> Native library checking:
> hadoop: true /opt/hadoop/hadoop-2.5.2/lib/native/libhadoop.so.1.0.0
> zlib:   true /lib64/libz.so.1
> snappy: true /opt/snappy/lib/libsnappy.so.1
> lz4:    true revision:99
> bzip2:  false‍
>
> But when I ran a MapReduce Job, it reported the following error:
> Error: java.lang.RuntimeException: native snappy library not available:
> SnappyCompressor has not been loaded.
>     at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:69)
>     at
> org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:143)
>     at
> org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:98)
>     at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:136)
> ‍
>
> I also tried to set io.compression.codecs, but it did not work too.
>
> The only way I found worked is to make a soft link as following:
> ln -s /opt/snappy/lib/libsnappy.so.1.2.1
> /opt/hadoop/hadoophome/lib/native/libsnappy.so.1
>
> I used to config snappy in Hadoop2.4.0 successfully. I remembered that I
> only need to config the LD_LIBRARY_PATH in /etc/profile. There is no need
> to make a such soft link. Does Hadoop2.5.2 not support this configuration
> anymore? Or is there other ways to config in Hadoop2.5.2 which don't
> require to make links or have to copy the lib to the hadoop's lib/native
> directory?
>
> Many Thanks!
>

Mime
View raw message