oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Resource Manager client question
Date Tue, 08 May 2012 13:58:43 GMT
Hi Mike,

I'm going to paste some of the conversation that you and I had below earlier b/c I think it's
of use to the list in general:

On May 7, 2012, at 3:43 PM, Iwunze, Michael C (GSFC-4700)[NOAA-JPSS] wrote:

> 
> I have two questions, I am able to run the Resource Manager with no issues. I have some
python scripts and possibly some other programs  I would like to run using the Resource Manager.
From what I know so far I believe the cas-pge component needs to be used in conjunction with
the Resource Manager and is used as a wrapper program for running my scripts. Can someone
give me more information on how this can be accomplished or are there any examples to view?


OK, probably the best way to do this would be to actually write a CAS-PGE workflow, and then
plug workflow manager into the Resource
Manager. There is a pretty good guide for this on the Apache OODT wiki, here:

https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example

Here is the link to the whole wiki:

https://cwiki.apache.org/confluence/display/OODT

In general the documentation on the Apache OODT wiki is way more up to date than the website
which typically only is updated
when we make a release. 

> In addition, I would like to be able to modify or extend the job scheduler and queue
for managing execution of my scripts. Do you know of any examples of something similar to
what I am trying to accomplish? 

Sure do, and would be happy to share. Check out CAS-PGE above for the running your custom
scripts. As for modifying the job
scheduler and the queue manager, please find the default implementations of these extensions
points, here:

http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/LRUScheduler.java

and here:

http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/LRUQueueManager.java

As for other implementations of them, I have always wanted to create the following:

Torque-based Queue implementation that talks to qsub
Monitor implementation that talks to Ganglia
Batch Mgr implementation that talks to Gexec
Batch Mgr implementation that talks to Torque

You can see some of these issues here:

https://issues.apache.org/jira/browse/OODT-231

And here:

https://issues.apache.org/jira/browse/OODT-219

I've been wanting to implement the Ganglia extension since 2005 -- when I was working on the
first
OCO mission. It's got an easy-to-access, socket/XML based interface that we could write a
parser to.
Hadoop did this and plugged into Ganglia via its metrics package, so we could probably look
at that.

HTH and let me know what you think!

> 
> I would also like to be able to utilize the Job Scheduler, Monitor and Job queue classes
that are part of the Resource Manager. I can't find any examples of how they are used anywhere.
And if examples do exist can someone point me in the right direction or give me more information
on this?

The Scheduler is the main point of entry, and it's used in the XmlRpcResourceManager class,
here:

http://s.apache.org/scU

Scheduler calls Monitor, as well as JobQueue, and is the nexus of interaction there.

HTH get you started!

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mime
View raw message