hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <gourav.had...@gmail.com>
Subject Re: setting LZO in Ubuntu
Date Tue, 24 Jul 2012 19:49:43 GMT
Hi,

I found out that copying the files from native folder to
/usr/local/hadoop/lib folder solved the problem but the main issue then is
why hadoop is not able to pick up the native libraries based on the
configuration of environment variables; i.e, why is java.library.path not
set from LD_LIBRARY_PATH or JAVA_LIBRARY_PATH  which are set to the folder
/usr/local/hadoop/lib/native/Linux-i386-32?

Kindly note that I am using Ubuntu 12.04 and there may be a difference in
the way it handles the LD_LIBRARY_PATH environment variables but this is
not the case since SQOOP is able to use the environment variable
LD_LIBRARY_PATH for locating Oracle JDBC jar files.

I will be grateful if someone could kindly point out what I am doing wrong
here, though there is a work around for now.

Thanks and Regards,
Gourav

On Tue, Jul 24, 2012 at 5:49 PM, Gourav Sengupta <gourav.hadoop@gmail.com>wrote:

> Hi,
>
> The HADOOP version is 0.20.205 and the jar file is hadoop-lzo-0.4.15.jar.
>
> I took the following steps in order to enable LZO compression in Ubuntu
> HADOOP:
> 1. sudo apt-get install liblzo2-dev
>
> 2. git clone https://github.com/kevinweil/hadoop-lzo.git
>
> 3. cd hadoop-lzo
>
> 4. changed the build.xml file and included the line <classpath
> refid="classpath"/> in <javah
> classpath="${build.classes}".................> tag
>
> 5. ran the following commands:
>         export CFLAGS=-m32
>         export CXXFLAGS=-m32
>         ant compile-native
>         ant jar
>
> 6. copied the file hadoop-lzo-0.4.15.jar to /usr/local/hadoop/lib
>
> 7. tar -cBf  - -C ./build/hadoop-lzo-0.4.15/lib/native  . | tar -xBvf - -C
> $HADOOP/lib/native
>
> 8. Added the following to core-site.xml:
> <property>
> <name>io.compression.codecs</name>
> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
>
> </property>
> <property>
> <name>io.compression.codec.lzo.class</name>
>  <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
>
> 9. Added the following line to mapred-site,xml
> <property>
>  <name>mapred.child.env</name>
> <value>JAVA_LIBRARY_PATH=/path/to/your/native/hadoop-lzo/libs</value>
> </property>
>
> 10. Modified $HADOOP/bin/hadoop so that the line JAVA_LIBRARY_PATH='' is
> commented
>
> 11. created the following environmental variables in hadoop-env.sh
> export
> JAVA_LIBRARY_PATH=/usr/local/hadoop/lib/native/Linux-i386-32:/usr/local/hadoop/lib:/usr/local/hadoop/lib/hadoop-lzo-0.4.15.jar
> export
> LD_LIBRARY_PATH=/usr/local/oraclejdbcconnector:/usr/local/hadoop/lib/native/Linux-i386-32:/usr/lib/i386-linux-gnu:/usr/local/hadoop/lib/hadoop-lzo-0.4.15.jar
> export
> HADOOP_CLASSPATH=$HADOOP/lib:/usr/local/hadoop/lib/hadoop-lzo-0.4.15.jar
>
> 12. restarted HADOOP
>
> 13. On running the command "hadoop jar
> /usr/local/hadoop/lib/hadoop-lzo-0.4.15.jar
> com.hadoop.compression.lzo.LzoIndexer /tmp/test.log.lzo" I am getting the
> following error:
> ERROR lzo.GPLNativeCodeLoader: Could not load native gpl
> libraryjava.lang.UnsatisfiedLinkError: no gplcompression in
> java.library.path
>
>
> I will be grateful if someone could kindly help me in resolving the issue.
>
>
> Thanks and Regards,
> Gourav
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message