hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Dunning" <ted.dunn...@gmail.com>
Subject Re: Is Hadoop the thing for us ?
Date Wed, 25 Jun 2008 15:49:43 GMT
This can work pretty well if you just use the list of parameter settings as
input.  The map task would run your simulation and output the data.  You may
not even need a reducer, although parallelized summary of output might be
very nice to have.  Because each of your sims takes a long time to run,
hadoop should be very efficient.

The only change you should need to make is to write a map class that
launches your simulation and copies whatever output you want into HDFS
instead of the local file system.  If you can get your sim to write to HDFS
directly that would be better.

On Wed, Jun 25, 2008 at 4:33 AM, Igor Nikolic <i.nikolic@tudelft.nl> wrote:

> Hello list
> We will be getting access to a cluster soon, and I was wondering whether
> this I should use Hadoop ?  Or am I better of with the usual batch
> schedulers such as ProActive etc ? I am not a CS/CE person, and from reading
> the website I can not get a sense of whether hadoop is for me.
> A little background:
> We have a  relatively large agent based simulation ( 20+ MB jar) that needs
> to be swept across very large parameter spaces. Agents communicate only
> within the simulation, so there is no interprocess communication. The
> parameter vector is max 20 long , the simulation may take 5-10 minutes on a
> normal desktop and it might return a few mb of raw data. We need 10k-100K
> runs, more if possible.
> Thanks for advice, even a short yes/no is welcome
> Greetings
> Igor
> --
> ir. Igor Nikolic
> PhD Researcher
> Section Energy & Industry
> Faculty of Technology, Policy and Management
> Delft University of Technology, The Netherlands
> Tel: +31152781135
> Email: i.nikolic@tudelft.nl
> Web: http://www.igornikolic.com
> wiki server: http://wiki.tudelft.nl


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message