hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arthur.hk.chan@gmail.com" <arthur.hk.c...@gmail.com>
Subject Hadoop 2.4.1 Snappy Smoke Test failed
Date Tue, 19 Aug 2014 15:40:53 GMT
Hi,

I am trying Snappy in Hadoop 2.4.1, here are my steps: 

(CentOS 64-bit)
1)
yum install snappy snappy-devel

2)
added the following 
(core-site.xml)
   <property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
   </property>

3) 
mapred-site.xml
   <property>
    <name>mapreduce.admin.map.child.java.opts</name>
    <value>-server -XX:NewRatio=8 -Djava.library.path=/usr/lib/hadoop/lib/native/ -Djava.net.preferIPv4Stack=true</value>
    <final>true</final>
   </property>
   <property>
    <name>mapreduce.admin.reduce.child.java.opts</name>
    <value>-server -XX:NewRatio=8 -Djava.library.path=/usr/lib/hadoop/lib/native/ -Djava.net.preferIPv4Stack=true</value>
    <final>true</final>
   </property>

4) smoke test
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar  teragen 100000 /tmp/teragenout

I got the following warning, actually there is no any test file created in hdfs:

14/08/19 22:50:10 WARN mapred.YARNRunner: Usage of -Djava.library.path in mapreduce.admin.map.child.java.opts
can cause programs to no longer function if hadoop native libraries are used. These values
should be set as part of the LD_LIBRARY_PATH in the map JVM env using mapreduce.admin.user.env
config settings.
14/08/19 22:50:10 WARN mapred.YARNRunner: Usage of -Djava.library.path in mapreduce.admin.reduce.child.java.opts
can cause programs to no longer function if hadoop native libraries are used. These values
should be set as part of the LD_LIBRARY_PATH in the reduce JVM env using mapreduce.admin.user.env
config settings.

Can anyone please advise how to install and enable SNAPPY in Hadoop 2.4.1? or what would be
wrong? or is my new change in mapred-site.xml incorrect?

Regards
Arthur






Mime
View raw message