oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (398J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Workflow Questions
Date Thu, 04 Apr 2013 16:54:21 GMT
Hey Mike,


-----Original Message-----
From: <Cayanan>, "Michael D   (388J)" <michael.d.cayanan@jpl.nasa.gov>
Reply-To: "user@oodt.apache.org" <user@oodt.apache.org>
Date: Tuesday, April 2, 2013 8:32 AM
To: "user@oodt.apache.org" <user@oodt.apache.org>
Subject: Re: Workflow Questions

>Hey Chris,
>
>I suppose I could. For some reason I didn't think that it was possible to
>submit a job to the Resource Manager other than through the Workflow.
>In my case, I'm using a listener to pull files out of the RabbitMQ. After
>I pull a file out of the queue, I'd like to submit a job to the Resource
>Manager to trigger a Workflow Event. How would one submit a job this way?

You just use the XmlRpcResourceManagerClient class (if you are writing a
Java
program to pull the file out of RabbitMQ).

If you are pulling out of RabbitMQ via a script, then you can use the
JobSubmitter
program that ships with Resource Manager. The job config file that you
mention:

>Under the Resource Manager trunk, in src/main/resources/examples/jobs, I
>see 2 xml files that appear to be config files. Do I need to make use of
>these to do what I want to do?

Is an XML specification of a job that you can pass to JobSubmitter. It
really
takes these input:

* JobInput formatter (Serializer/DeSerializer class for reading and writing
Job Input)
* JobClass
* config

That's about it.

>
>Also, I know that the Resource Manager has it's own queue where it places
>jobs when all of its compute nodes are full. In my case (this is for
>Costin's pipeline BTW), I don't want jobs being placed in the Resource
>Manager queue. I would like for them to remain in the Rabbit Queue. Is the
>Resource Manager's "getNodeLoad()" method the best way to check if a node
>is full before submitting a job?

Yep.

Cheers,
Chris

>
>Thanks in advance for your help!
>
>-Mike
>
>
>On 4/1/13 10:15 PM, "Mattmann, Chris A (388J)"
><chris.a.mattmann@jpl.nasa.gov> wrote:
>
>>Hey Mike,
>>
>>I'll bite, can't you use the Resource Manager and its queues/nodes to do
>>this?
>>
>>Cheers,
>>Chris
>>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Chris Mattmann, Ph.D.
>>Senior Computer Scientist
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 171-266B, Mailstop: 171-246
>>Email: chris.a.mattmann@nasa.gov
>>WWW:  http://sunset.usc.edu/~mattmann/
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Adjunct Assistant Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 90089 USA
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>>
>>-----Original Message-----
>>From: <Cayanan>, "Michael D   (388J)" <michael.d.cayanan@jpl.nasa.gov>
>>Reply-To: "user@oodt.apache.org" <user@oodt.apache.org>
>>Date: Monday, April 1, 2013 8:11 PM
>>To: "user@oodt.apache.org" <user@oodt.apache.org>
>>Subject: Workflow Questions
>>
>>>Hi all,
>>>
>>>
>>>I'm trying to see if anyone knows of a cool, easy to use "Resource
>>>Manager" that can control the amount of Workflows running concurrently
>>>on
>>>a node. Basically, my set up is like this:
>>>
>>>
>>>- I have a Workflow Manager Server configured to run at most 10 Workflow
>>>Events concurrently.
>>>- I have a bunch of files sitting on a RabbitMQ queue.
>>>- I have a listener that is continually monitoring this particular
>>>queue.
>>>As soon as it sees a file in this queue, it will trigger a Workflow
>>>event.
>>>
>>>
>>>Under this scenario, normally if I have 30 files in the RabbitMQ, my
>>>listener will trigger 30 Workflow events. However, the Workflow Server
>>>would put 20 of these events in its repository queue while the other 10
>>>are running.
>>>
>>>
>>>What I want some "Resource Manager" component to do is just run the 10
>>>Workflow events only and have the other 20 sitting in the RabbitMQ. As
>>>soon as 1 event has finished, then trigger another workflow event to
>>>keep
>>>the number of concurrent workflows running
>>> at 10.
>>>Eventually, we will want this "Resource Manager" component to be able to
>>>point to another node, node 2, where it can decide to trigger another
>>>set
>>>of 10 Workflow events while node 1 is still running it's 10 Workflow
>>>events.
>>>
>>>
>>>Hope this makes sense. Any helpful tips would be much appreciated.
>>>
>>>
>>>Thanks,
>>>Mike
>>
>

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mime
View raw message