hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hanish Bansal <hanish.bansal.agar...@gmail.com>
Subject Re: Hadoop 2.4 + Hive 0.14 + Hbase 0.98.3 + snappy not working
Date Thu, 11 Dec 2014 03:46:57 GMT
Hope this may help you:

http://blogs.impetus.com/big_data/big_data_technologies/SnappyCompressionInHBase.do

On Thu, Dec 11, 2014 at 7:25 AM, Fabio <anytek88@gmail.com> wrote:

> Plain Apache Hadoop 2.5.0.
> Too bad it didn't work, hope someone can help.
>
>
> On 12/10/2014 06:22 PM, peterm_second wrote:
>
>> Hi Fabio ,
>> Thanks for the reply, but unfortunately it didn't work. I am using
>> vanilla hadoop 2.4 with vanilla hive 0.14 and so on, I am using the vanilla
>> distros.
>> I did set the HADOOP_COMMON_LIB_NATIVE_DIR but that didn't make any
>> change. What version were you using ?
>>
>> Peter
>>
>>
>> On 10.12.2014 16:23, Fabio wrote:
>>
>>> Not sure it will help, but if the problem is native library loading, I
>>> spent a loooong time trying anything to make it work.
>>> I may suggest to try also:
>>> export JAVA_LIBRARY_PATH=/opt/yarn/hadoop-2.5.0/lib/native
>>> export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/yarn/hadoop-2.5.0/lib
>>> I have this both in the bash "init" script (/etc/profile.p/...) and in
>>> /opt/yarn/hadoop-2.5.0/etc/hadoop/hadoop-env.sh; quite sure it's
>>> redundant, but as long as it works I don't change it.
>>> I see here I commented out my attempts to set HADOOP_OPTS, so maybe it's
>>> not necessary.
>>> I don't see anything in my .xml config files.
>>> Also, someone says to compile the libraries under your 64 bit system,
>>> since the ones in Hadoop are for a 32bit architecture.
>>>
>>> Good luck
>>>
>>> Fabio
>>>
>>> On 12/10/2014 02:57 PM, peterm_second wrote:
>>>
>>>> Hi guys,
>>>> I have a hadoop + hbase + hive application,
>>>> For some reason my cluster is unable to find the snappy native library
>>>> Here is the exception :
>>>>  org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>>>>     at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
>>>> Method)
>>>>     at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
>>>>
>>>>     at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
>>>>
>>>>     at org.apache.hadoop.io.compress.CodecPool.getCompressor(
>>>> CodecPool.java:148)
>>>>     at org.apache.hadoop.io.compress.CodecPool.getCompressor(
>>>> CodecPool.java:163)
>>>>     at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115)
>>>>     at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.
>>>> sortAndSpill(MapTask.java:1583)
>>>>     at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(
>>>> MapTask.java:1462)
>>>>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
>>>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>>>>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>>     at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>     at org.apache.hadoop.security.UserGroupInformation.doAs(
>>>> UserGroupInformation.java:1548)
>>>>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>>>>
>>>>
>>>> I am working with a 64bit ubuntu 14.04LTS. I've installed snappy on my
>>>> os and added the coppied the libs to hadoop_home/lib/native
>>>> I've also added the libs to the JRE, but it still fails as if nothing
>>>> is present.
>>>> I've added  HADOOP_OPTS="-Djava.net.preferIPv4Stack=true
>>>> $GC_DEBUG_OPTS -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native
>>>> $HADOOP_OPTS"
>>>> in my yarn xml I have
>>>> <property>
>>>>       <name>yarn.app.mapreduce.am.env</name>
>>>> <value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>
>>>> </property>
>>>>
>>>> in my mapred-site.xml i have
>>>> <property>
>>>>         <name>mapred.child.java.opts</name>
>>>>         <value> -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native<
>>>> /value>
>>>>     </property>
>>>>     <property>
>>>>         <name>mapreduce.reduce.java.opts</name>
>>>> <value>-Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value>
>>>>     </property>
>>>>
>>>> The last two were a desperation move.
>>>> The result is always the same. Any ideas would be welcomed.
>>>>
>>>> Thanks,
>>>> Peter
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>> .
>>
>>
>


-- 
*Thanks & Regards*
*Hanish Bansal*

Mime
View raw message