hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tsuyoshi Ozawa <oz...@apache.org>
Subject Re: Something about the Snappy Compression Tool
Date Tue, 24 Mar 2015 10:35:11 GMT
Sometimes compiled native libraries included in tar ball doesn't work
correctly - how about recompiling library on your environment?

Thanks,
- Tsuyoshi

On Tue, Mar 24, 2015 at 6:18 PM, 王鹏飞 <wpf5511@gmail.com> wrote:
>         I noticed a map-reduce job encountered an
> Exception:java.lang.UnsatisfiedLinkError:org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>         I googled it  and released that it was lack of snappy tool.I
> Installed the snappy and copied libsnappy.so* to $HADOOP_HOME/lib/native
>         In .bashrc added:
>               export LD_LIBRARY_PATH="$HADOOP_HOME/lib/native/"
>               export JAVA_LIBRARY_PATH="$HADOOP_HOME/lib/native/"
>               export
> HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
>         In core-site.xml added:
>                        <property>
>                                <name>io.compression.codecs</name>
>                                       <value>
>
> org.apache.hadoop.io.compress.GzipCodec,
>
> org.apache.hadoop.io.compress.DefaultCodec,
>
> org.apache.hadoop.io.compress.BZip2Codec,
>
> org.apache.hadoop.io.compress.SnappyCodec
>                                        </value>
>                           </property>
>         In mapred-site.xml added:
>                              <property>
>                                  <name>mapreduce.map.output.compress</name>
>                                   <value>true</value>
>                             </property>
>
>                            <property>
>                               <name>mapred.map.output.compress.codec</name>
>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
>                           </property>
>
>                           <property>
>                                <name>mapreduce.admin.user.env</name>
>
> <value>LD_LIBRARY_PATH=/home/hadoop/software/hadoop-2.4.1/lib/native</value>
>                            </property>
>         In yarn-site.xml added:
>                   <property>
>                         <name>yarn.app.mapreduce.am.env</name>
>
> <value>LD_LIBRARY_PATH=/home/hadoop/software/hadoop-2.4.1/lib/native</value>
>                  </property>
>          Finally make the same setting in the datanode ,but it still didn't
> work 。But in a Pseudo-Distributed cluster, I only copy the libsnappy.so* to
> $HADOOP_HOME/lib/native without changing the configuration ,it works well
> .Why in  this cluster it  appears so difficult?
>
>
>
>
>
>

Mime
View raw message