hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From peterm_second <regest...@gmail.com>
Subject Re: Hadoop 2.4 + Hive 0.14 + Hbase 0.98.3 + snappy not working
Date Wed, 10 Dec 2014 17:39:05 GMT
Sorry I should've mentioned this. I've installed snappy lib using 
apt-get , my hadoop had no snappy  support build in.

Peter


On 10.12.2014 19:28, Ted Yu wrote:
> See:
> https://issues.apache.org/jira/browse/HADOOP-9911
>
> Can you recompile snappy for 64-bit system ?
>
> Cheers
>
> On Wed, Dec 10, 2014 at 9:22 AM, peterm_second <regestrer@gmail.com 
> <mailto:regestrer@gmail.com>> wrote:
>
>     Hi Fabio ,
>     Thanks for the reply, but unfortunately it didn't work. I am using
>     vanilla hadoop 2.4 with vanilla hive 0.14 and so on, I am using
>     the vanilla distros.
>     I did set the HADOOP_COMMON_LIB_NATIVE_DIR but that didn't make
>     any change. What version were you using ?
>
>     Peter
>
>
>
>     On 10.12.2014 16:23, Fabio wrote:
>
>         Not sure it will help, but if the problem is native library
>         loading, I spent a loooong time trying anything to make it work.
>         I may suggest to try also:
>         export JAVA_LIBRARY_PATH=/opt/yarn/hadoop-2.5.0/lib/native
>         export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/yarn/hadoop-2.5.0/lib
>         I have this both in the bash "init" script
>         (/etc/profile.p/...) and in
>         /opt/yarn/hadoop-2.5.0/etc/hadoop/hadoop-env.sh; quite sure
>         it's redundant, but as long as it works I don't change it.
>         I see here I commented out my attempts to set HADOOP_OPTS, so
>         maybe it's not necessary.
>         I don't see anything in my .xml config files.
>         Also, someone says to compile the libraries under your 64 bit
>         system, since the ones in Hadoop are for a 32bit architecture.
>
>         Good luck
>
>         Fabio
>
>         On 12/10/2014 02:57 PM, peterm_second wrote:
>
>             Hi guys,
>             I have a hadoop + hbase + hive application,
>             For some reason my cluster is unable to find the snappy
>             native library
>             Here is the exception :
>              org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>                 at
>             org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
>             Method)
>                 at
>             org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
>
>                 at
>             org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
>
>                 at
>             org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>                 at
>             org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
>                 at
>             org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115)
>                 at
>             org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1583)
>
>                 at
>             org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462)
>                 at
>             org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
>                 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>                 at
>             org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>                 at java.security.AccessController.doPrivileged(Native
>             Method)
>                 at javax.security.auth.Subject.doAs(Subject.java:422)
>                 at
>             org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
>                 at
>             org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
>
>             I am working with a 64bit ubuntu 14.04LTS. I've installed
>             snappy on my os and added the coppied the libs to
>             hadoop_home/lib/native
>             I've also added the libs to the JRE, but it still fails as
>             if nothing is present.
>             I've added  HADOOP_OPTS="-Djava.net.preferIPv4Stack=true
>             $GC_DEBUG_OPTS
>             -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native
>             $HADOOP_OPTS"
>             in my yarn xml I have
>             <property>
>                   <name>yarn.app.mapreduce.am
>             <http://yarn.app.mapreduce.am>.env</name>
>             <value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>
>             </property>
>
>             in my mapred-site.xml i have
>             <property>
>                     <name>mapred.child.java.opts</name>
>                     <value>
>             -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value>
>                 </property>
>                 <property>
>                     <name>mapreduce.reduce.java.opts</name>
>             <value>-Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value>
>                 </property>
>
>             The last two were a desperation move.
>             The result is always the same. Any ideas would be welcomed.
>
>             Thanks,
>             Peter
>
>
>
>
>
>
>
>
>
>


Mime
View raw message