oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cameron Goodale <sigep...@gmail.com>
Subject Re: Resource Manager Info.
Date Wed, 18 Apr 2012 14:05:21 GMT
Hey Mike,

I don't have any formal documentation on the Resource Manager, but I can
try to explain how the Snow Data System team is using it to process large
amounts of MODIS MOD09GA tile data.  From there it might be easier to just
ask additional questions since the Components are a lot like legos and you
can swap them around to make several different systems in the end.

On Snow (Snow Data System Project) we have 5 Servers (1 VM, and 4 Work
Horses with 48 Cores Each).  The Workflow Manager and Resource Manager are
installed on the VM and it is sort of like a head node where it does all
the Job Creation and Submission, but it does none of the tile processing.
 Here is a list of high-level setups/config that enable us to run the
MOSCAG algorithm across 3 of the 4 Work Horse Machines (the 4th Machine is
used for another processing pipeline, so we will ignore it for now).


   1. On the Head Node (VM) we have a Workflow Manager (WM) configured to
   Create MODSCAG Workflows.
   2. On the Head Node we also have a Resource Manager (RM) configured to
   send jobs out to BatchStubs on the 3 Work Horse Machines.
   3. On the Head Node the WM is configured to USE the RM.  This means the
   WM will submit jobs to the RM until the RM has reached Capacity, then the
   Workflow Manager will queue any additional jobs and wait for the RM to have
   additional Capacity by completing a Job.
   4. Our MODSCAG Jobs run in a Quad Core Process, so we have configured
   the RM to only submit 10 Jobs to each Work Horse machine (hence using 40 of
   the 48 cores).  This is unique to the image processing code base, your
   mileage (and core usage) may vary.

So I hope this helps.  Again your configuration might be that you have 100
4-core machines that you want the RM to use for job processing, or you may
want to stand up BatchStubs on your localhost for testing purposes.  At
that point it really is just a matter of getting the configuration setup
and restarting the components.

If anyone else in the community has anything to add or comment, that would
be great.  And Mike if you have any questions at all, don't hesitate to ask.

Best Regards,


Cameron


On Tue, Apr 17, 2012 at 9:52 AM, Iwunze, Michael C (GSFC-4700)[NOAA-JPSS] <
michael.iwunze@nasa.gov> wrote:

> Hello,
>
>         I am one of the GRAVITE developers that is new to OODT. Currently
> researching the resource manager component, I saw the user guide online. Do
> you know of any other locations were I can get more information on this
> component as well as more examples and how it can be used?
>
> Thanks
>
> Mike
>



-- 

Sent from a Tin Can attached to a String

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message