hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From peterm_second <regest...@gmail.com>
Subject Re: Hadoop 2.4 + Hive 0.14 + Hbase 0.98.3 + snappy not working
Date Wed, 10 Dec 2014 17:22:14 GMT
Hi Fabio ,
Thanks for the reply, but unfortunately it didn't work. I am using 
vanilla hadoop 2.4 with vanilla hive 0.14 and so on, I am using the 
vanilla distros.
I did set the HADOOP_COMMON_LIB_NATIVE_DIR but that didn't make any 
change. What version were you using ?

Peter


On 10.12.2014 16:23, Fabio wrote:
> Not sure it will help, but if the problem is native library loading, I 
> spent a loooong time trying anything to make it work.
> I may suggest to try also:
> export JAVA_LIBRARY_PATH=/opt/yarn/hadoop-2.5.0/lib/native
> export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/yarn/hadoop-2.5.0/lib
> I have this both in the bash "init" script (/etc/profile.p/...) and in 
> /opt/yarn/hadoop-2.5.0/etc/hadoop/hadoop-env.sh; quite sure it's 
> redundant, but as long as it works I don't change it.
> I see here I commented out my attempts to set HADOOP_OPTS, so maybe 
> it's not necessary.
> I don't see anything in my .xml config files.
> Also, someone says to compile the libraries under your 64 bit system, 
> since the ones in Hadoop are for a 32bit architecture.
>
> Good luck
>
> Fabio
>
> On 12/10/2014 02:57 PM, peterm_second wrote:
>> Hi guys,
>> I have a hadoop + hbase + hive application,
>> For some reason my cluster is unable to find the snappy native library
>> Here is the exception :
>>  org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>>     at 
>> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native 
>> Method)
>>     at 
>> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)

>>
>>     at 
>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)

>>
>>     at 
>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>>     at 
>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>>     at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115)
>>     at 
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1583)

>>
>>     at 
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462)
>>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:422)
>>     at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

>>
>>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>>
>>
>> I am working with a 64bit ubuntu 14.04LTS. I've installed snappy on 
>> my os and added the coppied the libs to hadoop_home/lib/native
>> I've also added the libs to the JRE, but it still fails as if nothing 
>> is present.
>> I've added  HADOOP_OPTS="-Djava.net.preferIPv4Stack=true 
>> $GC_DEBUG_OPTS -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native 
>> $HADOOP_OPTS"
>> in my yarn xml I have
>> <property>
>>       <name>yarn.app.mapreduce.am.env</name>
>> <value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>
>> </property>
>>
>> in my mapred-site.xml i have
>> <property>
>>         <name>mapred.child.java.opts</name>
>>         <value> 
>> -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value>
>>     </property>
>>     <property>
>>         <name>mapreduce.reduce.java.opts</name>
>> <value>-Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value>
>>     </property>
>>
>> The last two were a desperation move.
>> The result is always the same. Any ideas would be welcomed.
>>
>> Thanks,
>> Peter
>>
>>
>>
>>
>>
>>
>>
>


Mime
View raw message