hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王鹏飞 <wpf5...@gmail.com>
Subject Something about the Snappy Compression Tool
Date Tue, 24 Mar 2015 09:18:36 GMT
        I noticed a map-reduce job encountered an
Exception:java.lang.UnsatisfiedLinkError:org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
        I googled it  and released that it was lack of snappy tool.I
Installed the snappy and copied libsnappy.so* to $HADOOP_HOME/lib/native
        In .bashrc added:
              export LD_LIBRARY_PATH="$HADOOP_HOME/lib/native/"
              export JAVA_LIBRARY_PATH="$HADOOP_HOME/lib/native/"
              export
HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
        In core-site.xml added:
                       <property>
                               <name>io.compression.codecs</name>
                                      <value>

org.apache.hadoop.io.compress.GzipCodec,

org.apache.hadoop.io.compress.DefaultCodec,

org.apache.hadoop.io.compress.BZip2Codec,

org.apache.hadoop.io.compress.SnappyCodec
                                       </value>
                          </property>
        In mapred-site.xml added:
                             <property>
                                 <name>mapreduce.map.output.compress</name>
                                  <value>true</value>
                            </property>

                           <property>
                              <name>mapred.map.output.compress.codec</name>

<value>org.apache.hadoop.io.compress.SnappyCodec</value>
                          </property>

                          <property>
                               <name>mapreduce.admin.user.env</name>

<value>LD_LIBRARY_PATH=/home/hadoop/software/hadoop-2.4.1/lib/native</value>
                           </property>
        In yarn-site.xml added:
                  <property>
                        <name>yarn.app.mapreduce.am.env</name>

<value>LD_LIBRARY_PATH=/home/hadoop/software/hadoop-2.4.1/lib/native</value>
                 </property>
         Finally make the same setting in the datanode ,but it still didn't
work 。But in a Pseudo-Distributed cluster, I only copy the libsnappy.so* to
$HADOOP_HOME/lib/native without changing the configuration ,it works well
.Why in  this cluster it  appears so difficult?

Mime
View raw message