hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabio <anyte...@gmail.com>
Subject Re: Hadoop 2.4 + Hive 0.14 + Hbase 0.98.3 + snappy not working
Date Wed, 10 Dec 2014 14:23:11 GMT
Not sure it will help, but if the problem is native library loading, I 
spent a loooong time trying anything to make it work.
I may suggest to try also:
export JAVA_LIBRARY_PATH=/opt/yarn/hadoop-2.5.0/lib/native
export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/yarn/hadoop-2.5.0/lib
I have this both in the bash "init" script (/etc/profile.p/...) and in 
/opt/yarn/hadoop-2.5.0/etc/hadoop/hadoop-env.sh; quite sure it's 
redundant, but as long as it works I don't change it.
I see here I commented out my attempts to set HADOOP_OPTS, so maybe it's 
not necessary.
I don't see anything in my .xml config files.
Also, someone says to compile the libraries under your 64 bit system, 
since the ones in Hadoop are for a 32bit architecture.

Good luck

Fabio

On 12/10/2014 02:57 PM, peterm_second wrote:
> Hi guys,
> I have a hadoop + hbase + hive application,
> For some reason my cluster is unable to find the snappy native library
> Here is the exception :
>  org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>     at 
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native 
> Method)
>     at 
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)

>
>     at 
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132) 
>
>     at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>     at 
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>     at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115)
>     at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1583) 
>
>     at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462)
>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:422)
>     at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

>
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
>
> I am working with a 64bit ubuntu 14.04LTS. I've installed snappy on my 
> os and added the coppied the libs to hadoop_home/lib/native
> I've also added the libs to the JRE, but it still fails as if nothing 
> is present.
> I've added  HADOOP_OPTS="-Djava.net.preferIPv4Stack=true 
> $GC_DEBUG_OPTS -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native 
> $HADOOP_OPTS"
> in my yarn xml I have
> <property>
>       <name>yarn.app.mapreduce.am.env</name>
> <value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>
> </property>
>
> in my mapred-site.xml i have
> <property>
>         <name>mapred.child.java.opts</name>
>         <value> 
> -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value>
>     </property>
>     <property>
>         <name>mapreduce.reduce.java.opts</name>
> <value>-Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value>
>     </property>
>
> The last two were a desperation move.
> The result is always the same. Any ideas would be welcomed.
>
> Thanks,
> Peter
>
>
>
>
>
>
>


Mime
View raw message