hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Kallestad <st...@tabtonic.com>
Subject RE: Unable to load native-hadoop library for your platform
Date Thu, 13 Feb 2014 05:35:18 GMT
Funny, I was just trying to add something to the wiki addressing this.



These instructions are for 2.2, but I imagine that 2.0.5 is probably very
similar.



If the formatting doesn't come through for whatever reason, I posted the
same thing here:



http://answers.splunk.com/answers/118174/hunk-reports-an-error-with-apache-hadoop?page=1&focusedAnswerId=122311#122311



This isn't necessarily a big problem - Hadoop will function without native
libraries.  You may find it easier to ignore the message or
disable/redirect JVM warnings.

You can disable the error message or redirect it to stderr, but that only
moves the error out of your way and doesn't deal with the root problem. The
root problem is that the hadoop distribution does not include native
libraries. They must be compiled from source.

You can build your own distribution that includes native libraries using
the following steps:

1) Install developer tools and dependencies:

1a) From repositories:

apt-get install gcc g++ make maven cmake zlib zlib1g-dev

for RedHat environments, you can probably use a similar yum line:

yum install gcc g++ make maven cmake zlib zlib-devel

There may be some other dependencies or slightly different package names
depending on what you already have installed and what OS you are running.
If so, some google-able errors will pop up during the rest of the process.

1b) Protocol Buffers From Source:

mkdir /tmp/protobuf

cd /tmp/protobuf

wget http:// protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

tar -xvzf ./protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

./configure --prefix=/usr

make

sudo make install

cd java

mvn install

mvn package

sudo ldconfig

cd /tmp

rm -rf protobuf

2) download hadoop source:

mkdir /tmp/hadoop-build

cd /tmp/hadoop-build

wget http://
apache.petsads.us/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz

tar -xvzf ./hadoop-2.2.0-src.tar.gz

cd hadoop-2.2.0-src

3) Edit the hadoop-auth pom file.

vi hadoop-common-project/hadoop-auth/pom.xml

add the following dependency:

<dependency>

   <groupId>org.mortbay.jetty</groupId>

  <artifactId>jetty-util</artifactId>

  <scope>test</scope>

</dependency>

You should see an already existing dependency that looks very similar if
you search for "org.mortbay.jetty", add this dependency above or below it.

3) Compile it:

export Platform=x64

cd /tmp/hadoop-build/hadoop-2.2.0-src

mvn clean install -DskipTests

cd hadoop-mapreduce-project

mvn package -Pdist,native -DskipTests=true -Dtar

cd /tmp/hadoop-build/hadoop-2.2.0-src

mvn package -Pdist,native -DskipTests=true -Dtar

4) Copy your natively compiled distribution somewhere to be saved:

cp
/tmp/hadoop-build/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0.tar.gz
/my/distribution/share/hadoop-2.2.0.tar.gz

5) Delete the build files (once you are satisfied that everything is
working properly):

cd /tmp

rm -rf hadoop-build

Now any fresh installations based on this build will include native 64 bit
libraries. You can set up a new instance of hadoop locally, or you can
simply overwrite the files in the $HADOOP-INSTALL/lib/native directory with
those in your hadoop-2.2.0.tar.gz file.





*From:* xeon Mailinglist [mailto:xeonmailinglist@gmail.com]
*Sent:* Wednesday, February 12, 2014 9:28 PM
*To:* user@hadoop.apache.org
*Subject:* Unable to load native-hadoop library for your platform



I am trying to run an example and I get the following error:



HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki

OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.

It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.

14/02/13 05:24:48 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable





I tried to run execstack -c, but the problem stays the same. Any help?

HadoopMaster-nh:~# execstack -c
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0

HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki

OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.

It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.

14/02/13 05:26:45 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable

Mime
View raw message