hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From peterm_second <regest...@gmail.com>
Subject Hadoop 2.4 + Hive 0.14 + Hbase 0.98.3 + snappy not working
Date Wed, 10 Dec 2014 13:57:35 GMT
Hi guys,
I have a hadoop + hbase + hive application,
For some reason my cluster is unable to find the snappy native library
Here is the exception :
  org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
     at 
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
     at 
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) 

     at 
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132) 

     at 
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
     at 
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
     at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115)
     at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1583) 

     at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462)
     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:422)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) 

     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)


I am working with a 64bit ubuntu 14.04LTS. I've installed snappy on my 
os and added the coppied the libs to hadoop_home/lib/native
I've also added the libs to the JRE, but it still fails as if nothing is 
present.
I've added  HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $GC_DEBUG_OPTS 
-Djava.library.path=/usr/local/hadoop-2.4.0/lib/native $HADOOP_OPTS"
in my yarn xml I have
<property>
       <name>yarn.app.mapreduce.am.env</name>
<value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>
</property>

in my mapred-site.xml i have
<property>
         <name>mapred.child.java.opts</name>
         <value> 
-Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value>
     </property>
     <property>
         <name>mapreduce.reduce.java.opts</name>
<value>-Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value>
     </property>

The last two were a desperation move.
The result is always the same. Any ideas would be welcomed.

Thanks,
Peter







Mime
View raw message