hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stijn De Weirdt <stijn.dewei...@ugent.be>
Subject Re: Hadoop-on-demand and torque
Date Sun, 20 May 2012 08:52:09 GMT
hi all,

i'm part of an HPC group of a university, and we have some users that 
are interested in Hadoop to see if it can be useful in their research 
and we also have researchers that are using hadoop already on their own 
infrastructure, but that is is not enough reason for us to start with 
dedicated dedicated Hadoop infrastructure  (we are now only running 
torque based clusters with and without shared storage; setting up and 
properly maintaining Hadoop infrastructure requires quite some 
understanding of new software)

to be able to support these needs we wanted to do just this: use current 
HPC infrastructure to make private hadoop clusters so people can do some 
work. if we attract enough interest, we will probably setup dedicated 
infrastructure, but by that time we (the admins) will also have a better 
understanding of what is required.

so we used to look at HOD for testing/running hadoop on existing 
infrastructure (never really looked at myhadoop though).
but (imho) the current HOD code base is not in such a good state. we did 
some work to get it working and added some features, to come to the 
conclusion that it was not sufficient (and not maintainable).

so we wrote something from scratch with same functionality as HOD, and 
much more (eg HBase is now possible, with or without MR1; some default 
tuning; easy to add support for yarn instead of MR1).
it has some suport for torque, but my laptop is also sufficient. (the 
torque support is a wrapper to submit the job)
we gave a workshop on hadoop using it (25 people, and each with their 
own 5 node hadoop cluster) and it went rather well.

it's not in a public repo yet, but we could do that. if interested, let 
me know, and i see what can be done. (releasing the code is on our todo 
list, but if there is some demand, we can do it sooner)


On 05/18/2012 05:07 PM, Pierre Antoine DuBoDeNa wrote:
> I am also interested to learn about myHadoop as I use a shared storage
> system and everything runs on VMs and not actual dedicated servers.
> in like amazon EC2 environment which you just have VMs and huge central
> storage, is it any helpful to use hadoop to distribute jobs and maybe
> parallelize algorithms, or is better to go with other technologies?
> 2012/5/18 Manu S<manupkd87@gmail.com>
>> Hi All,
>> Guess HOD could be useful existing HPC cluster with Torque scheduler which
>> needs to run map-reduce jobs.
>> Also read about *myHadoop- Hadoop on demand on traditional HPC
>> resources*will support many HPC schedulers like SGE, PBS etc to over
>> come the
>> integration of shared-architecture(HPC)&  shared-nothing
>> architecture(Hadoop).
>> Any real use case scenarios for integrating hadoop map/reduce in existing
>> HPC cluster and what are the advantages of using hadoop features in HPC
>> cluster?
>> Appreciate your comments on the same.
>> Thanks,
>> Manu S
>> On Fri, May 18, 2012 at 12:41 AM, Merto Mertek<masmertoz@gmail.com>
>> wrote:
>>> If I understand it right HOD is mentioned mainly for merging existing HPC
>>> clusters with hadoop and for testing purposes..
>>> I cannot find what is the role of Torque here (just initial nodes
>>> allocation?) and which is the default scheduler of HOD ?  Probably the
>>> scheduler from the hadoop distribution?
>>> In the doc is mentioned a MAUI scheduler, but probably if there would be
>> an
>>> integration with hadoop there will be any document on it..
>>> thanks..

View raw message