hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王鹏飞 <wpf5...@gmail.com>
Subject Re: Something about the Snappy Compression Tool
Date Wed, 25 Mar 2015 01:39:51 GMT
How to recompiling  library?Cause my hadoop was built from tar ball and in
/lib/native there exist files like *.a and *.so.Does I need build from
source code to recompiling library?

On Tue, Mar 24, 2015 at 6:35 PM, Tsuyoshi Ozawa <ozawa@apache.org> wrote:

> Sometimes compiled native libraries included in tar ball doesn't work
> correctly - how about recompiling library on your environment?
>
> Thanks,
> - Tsuyoshi
>
> On Tue, Mar 24, 2015 at 6:18 PM, 王鹏飞 <wpf5511@gmail.com> wrote:
> >         I noticed a map-reduce job encountered an
> >
> Exception:java.lang.UnsatisfiedLinkError:org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> >         I googled it  and released that it was lack of snappy tool.I
> > Installed the snappy and copied libsnappy.so* to $HADOOP_HOME/lib/native
> >         In .bashrc added:
> >               export LD_LIBRARY_PATH="$HADOOP_HOME/lib/native/"
> >               export JAVA_LIBRARY_PATH="$HADOOP_HOME/lib/native/"
> >               export
> > HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
> >         In core-site.xml added:
> >                        <property>
> >                                <name>io.compression.codecs</name>
> >                                       <value>
> >
> > org.apache.hadoop.io.compress.GzipCodec,
> >
> > org.apache.hadoop.io.compress.DefaultCodec,
> >
> > org.apache.hadoop.io.compress.BZip2Codec,
> >
> > org.apache.hadoop.io.compress.SnappyCodec
> >                                        </value>
> >                           </property>
> >         In mapred-site.xml added:
> >                              <property>
> >
> <name>mapreduce.map.output.compress</name>
> >                                   <value>true</value>
> >                             </property>
> >
> >                            <property>
> >
>  <name>mapred.map.output.compress.codec</name>
> >
> > <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> >                           </property>
> >
> >                           <property>
> >                                <name>mapreduce.admin.user.env</name>
> >
> >
> <value>LD_LIBRARY_PATH=/home/hadoop/software/hadoop-2.4.1/lib/native</value>
> >                            </property>
> >         In yarn-site.xml added:
> >                   <property>
> >                         <name>yarn.app.mapreduce.am.env</name>
> >
> >
> <value>LD_LIBRARY_PATH=/home/hadoop/software/hadoop-2.4.1/lib/native</value>
> >                  </property>
> >          Finally make the same setting in the datanode ,but it still
> didn't
> > work 。But in a Pseudo-Distributed cluster, I only copy the libsnappy.so*
> to
> > $HADOOP_HOME/lib/native without changing the configuration ,it works well
> > .Why in  this cluster it  appears so difficult?
> >
> >
> >
> >
> >
> >
>

Mime
View raw message