hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Azuryy Yu <azury...@gmail.com>
Subject Re: Something about the Snappy Compression Tool
Date Wed, 25 Mar 2015 04:07:16 GMT
Hi,
You should compile Hadoop source code with -Drequire.snappy, such as:
mvn -DskipTests -Pnative,dist -Dtar,require.snappy package

but you also need to install snappy before your compiling..



On Wed, Mar 25, 2015 at 9:39 AM, 王鹏飞 <wpf5511@gmail.com> wrote:

> How to recompiling  library?Cause my hadoop was built from tar ball and in
> /lib/native there exist files like *.a and *.so.Does I need build from
> source code to recompiling library?
>
> On Tue, Mar 24, 2015 at 6:35 PM, Tsuyoshi Ozawa <ozawa@apache.org> wrote:
>
>> Sometimes compiled native libraries included in tar ball doesn't work
>> correctly - how about recompiling library on your environment?
>>
>> Thanks,
>> - Tsuyoshi
>>
>> On Tue, Mar 24, 2015 at 6:18 PM, 王鹏飞 <wpf5511@gmail.com> wrote:
>> >         I noticed a map-reduce job encountered an
>> >
>> Exception:java.lang.UnsatisfiedLinkError:org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>> >         I googled it  and released that it was lack of snappy tool.I
>> > Installed the snappy and copied libsnappy.so* to $HADOOP_HOME/lib/native
>> >         In .bashrc added:
>> >               export LD_LIBRARY_PATH="$HADOOP_HOME/lib/native/"
>> >               export JAVA_LIBRARY_PATH="$HADOOP_HOME/lib/native/"
>> >               export
>> > HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
>> >         In core-site.xml added:
>> >                        <property>
>> >                                <name>io.compression.codecs</name>
>> >                                       <value>
>> >
>> > org.apache.hadoop.io.compress.GzipCodec,
>> >
>> > org.apache.hadoop.io.compress.DefaultCodec,
>> >
>> > org.apache.hadoop.io.compress.BZip2Codec,
>> >
>> > org.apache.hadoop.io.compress.SnappyCodec
>> >                                        </value>
>> >                           </property>
>> >         In mapred-site.xml added:
>> >                              <property>
>> >
>> <name>mapreduce.map.output.compress</name>
>> >                                   <value>true</value>
>> >                             </property>
>> >
>> >                            <property>
>> >
>>  <name>mapred.map.output.compress.codec</name>
>> >
>> > <value>org.apache.hadoop.io.compress.SnappyCodec</value>
>> >                           </property>
>> >
>> >                           <property>
>> >                                <name>mapreduce.admin.user.env</name>
>> >
>> >
>> <value>LD_LIBRARY_PATH=/home/hadoop/software/hadoop-2.4.1/lib/native</value>
>> >                            </property>
>> >         In yarn-site.xml added:
>> >                   <property>
>> >                         <name>yarn.app.mapreduce.am.env</name>
>> >
>> >
>> <value>LD_LIBRARY_PATH=/home/hadoop/software/hadoop-2.4.1/lib/native</value>
>> >                  </property>
>> >          Finally make the same setting in the datanode ,but it still
>> didn't
>> > work 。But in a Pseudo-Distributed cluster, I only copy the
>> libsnappy.so* to
>> > $HADOOP_HOME/lib/native without changing the configuration ,it works
>> well
>> > .Why in  this cluster it  appears so difficult?
>> >
>> >
>> >
>> >
>> >
>> >
>>
>
>

Mime
View raw message