hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Salman Toor <salman.t...@it.uu.se>
Subject Re: C++ example for hadoop-2.2.0
Date Wed, 06 Nov 2013 16:18:46 GMT
Hi, 

Sorry for the late response.

As I have mentioned earlier I am quite new to the system so have very primitive knowledge.


Basically I have followed the following blog: 

http://jugnu-life.blogspot.com/2013/09/build-and-compile-hadoop-from-source.html 

except the following line in the end 

# mvn package -Pdist -DskipTests -Dtar

I have used following command

# mvn package -Pdist,native,docs -DskipTests -Dtar

And than in "hadoop-dist/target" directory you can find "hadoop-2.2.0" which will contain
the right native libraries. 

Regards..
Salman. 




 



Salman Toor, PhD
salman.toor@it.uu.se



On Nov 5, 2013, at 9:42 PM, jerome wrote:

> Hi Salman
> 
> I've trying to compile too te obtain the good library. But i've fac some compile issue.
Could you please, send me how you compile the sources?
> 
> Reagards
> 
> On Tue, 5 Nov 2013 12:58:08 +0100, Salman Toor <salman.toor@it.uu.se> wrote:
>> Hi,
>> 
>> I have solved the problem! Have downloaded the source, compiled it on
>> the machine and now can successfully link to the library.
>> 
>> Thank you everyone for you quick responses.
>> 
>> Regards..
>> Salman.
>> 
>> Salman Toor, PhD
>> salman.toor@it.uu.se [1]
>> 
>> On Nov 4, 2013, at 3:21 PM, Andre Kelpe wrote:
>> 
>> No, because I was trying to set up a cluster automatically with the
>> tarballs from apache.org [2].
>> 
>> - André
>> 
>> On Mon, Nov 4, 2013 at 3:05 PM, Salman Toor  wrote:
>> Hi,
>> 
>> Did you tried to compile with source?
>> 
>> /Salman.
>> 
>> Salman Toor, PhD
>> salman.toor@it.uu.se [4]
>> 
>> On Nov 4, 2013, at 2:55 PM, Andre Kelpe wrote:
>> 
>> I reported the 32bit/64bit problem a few weeks ago. There hasn't been
>> much activity around it though:
>> https://issues.apache.org/jira/browse/HADOOP-9911 [5]
>> 
>> - André
>> 
>> On Mon, Nov 4, 2013 at 2:20 PM, Salman Toor  wrote:
>> 
>> Hi,
>> 
>> Ok so 2.x is not a new version its another branch. Good to know!
>> Actually
>> 
>> 32bit will be difficult as the code I got have already some
>> dependencies on
>> 
>> 64 bit.
>> 
>> Otherwise I will continue with 1.x version. Can you suggest some
>> version
>> 
>> with 1.x series which is stable and work on the cluster environment?
>> 
>> especially with C++ ...
>> 
>> Regards..
>> 
>> Salman.
>> 
>> Salman Toor, PhD
>> 
>> salman.toor@it.uu.se [7]
>> 
>> On Nov 4, 2013, at 1:54 PM, Amr Shahin wrote:
>> 
>> Well, the 2 series isn't exactly the "next version". It's a
>> continuation of
>> 
>> branch .2.
>> 
>> Also, the error message from the gcc indicates that the library
>> you're
>> 
>> trying to link to isn't compatible which made me suspicious. check
>> the
>> 
>> documentation to see if hadoop has 64 libraries, or otherwise compile
>> 
>> against the 32 ones
>> 
>> On Mon, Nov 4, 2013 at 4:51 PM, Salman Toor  wrote:
>> 
>> Hi,
>> 
>> Thanks for your answer!
>> 
>> But are you sure about it? Actually Hadoop version 1.2 have both 32
>> and 64
>> 
>> bit libraries so I believe the the next version should have both...
>> But I am
>> 
>> not sure just a random guess :-(
>> 
>> Regards..
>> 
>> Salman.
>> 
>> Salman Toor, PhD
>> 
>> salman.toor@it.uu.se [9]
>> 
>> On Nov 4, 2013, at 1:38 PM, Amr Shahin wrote:
>> 
>> I believe hadoop isn't compatible with 64 architecture. Try
>> installing the
>> 
>> 32 libraries and compile against them.
>> 
>> This error (skipping incompatible
>> 
>> /home/sztoor/hadoop-2.2.0/lib/native/libhadooppipes.a when searching
>> 
>> -lhadooppipes) indicates so
>> 
>> On Mon, Nov 4, 2013 at 2:44 PM, Salman Toor  wrote:
>> 
>> Hi,
>> 
>> Can someone give a pointer?
>> 
>> Thanks in advance.
>> 
>> Regards..
>> 
>> Salman.
>> 
>> Salman Toor, PhD
>> 
>> salman.toor@it.uu.se [11]
>> 
>> On Nov 3, 2013, at 11:31 PM, Salman Toor wrote:
>> 
>> Hi,
>> 
>> I am quite new to the Hadoop world, previously was running
>> hadoop-1.2.0
>> 
>> stable version on my small cluster and encountered some strange
>> problems
>> 
>> like the local path to the mapper file didn't copy to the hdfs ....
>> It
>> 
>> works fine on the single node setup but on multiple node simple
>> word-count
>> 
>> python example didn't work... I read on the blog that it might be the
>> 
>> problem in the version I am using. So I thought to change the version
>> and
>> 
>> downloaded the Hadoop 2.2.0. This version has yarn together with many
>> new
>> 
>> features that I hope I will learn in the future. Now simple wordcount
>> 
>> example works without any problem on the multi-node setup. I am using
>> simple
>> 
>> python example.
>> 
>> Now I would like to compile my C++ code. Since the directory
>> structure
>> 
>> together with other things have been changed. I have started to get
>> the
>> 
>> following error:
>> 
>> ----------------
>> 
>> /urs/bin/ld: skipping incompatible
>> 
>> /home/sztoor/hadoop-2.2.0/lib/native/libhadooputils.a when searching
>> 
>> -lhadooputils
>> 
>> cannot find -lhadooputils
>> 
>> /urs/bin/ld: skipping incompatible
>> 
>> /home/sztoor/hadoop-2.2.0/lib/native/libhadooppipes.a when searching
>> 
>> -lhadooppipes
>> 
>> cannot find -lhadooppipes
>> 
>> ------------------
>> 
>> I have managed to run the c++ example successfully with the 1.2.0
>> version
>> 
>> on single node setup.
>> 
>> I am having 64bit Ubuntu machine. previously I was using
>> Linux-amd64-64
>> 
>> Now in new version, "lib" and "include" directories are in the
>> 
>> hadoop-2.2.0 directory. No build.xml is available...
>> 
>> Can someone please give me an example of a makefile based on the
>> version
>> 
>> 2.2.0? Or suggest me which version I should go for? Or if there are
>> some
>> 
>> prerequisites that I should do before compiling my code?
>> 
>> Thanks in advance.
>> 
>> Regards..
>> 
>> Salman.
>> 
>> Salman Toor, PhD
>> 
>> salman.toor@it.uu.se [12]
>> 
>> --
>> André Kelpe
>> andre@concurrentinc.com [13]
>> http://concurrentinc.com [14]
>> 
>> --
>> André Kelpe
>> andre@concurrentinc.com [15]
>> http://concurrentinc.com
>> 
>> 
>> 
>> Links:
>> ------
>> [1] mailto:salman.toor@it.uu.se
>> [2] http://apache.org
>> [3] mailto:salman.toor@it.uu.se
>> [4] mailto:salman.toor@it.uu.se
>> [5] https://issues.apache.org/jira/browse/HADOOP-9911
>> [6] mailto:salman.toor@it.uu.se
>> [7] mailto:salman.toor@it.uu.se
>> [8] mailto:salman.toor@it.uu.se
>> [9] mailto:salman.toor@it.uu.se
>> [10] mailto:salman.toor@it.uu.se
>> [11] mailto:salman.toor@it.uu.se
>> [12] mailto:salman.toor@it.uu.se
>> [13] mailto:andre@concurrentinc.com
>> [14] http://concurrentinc.com
>> [15] mailto:andre@concurrentinc.com
> 


Mime
View raw message