oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (3980)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Request for advice on workflows
Date Sat, 16 May 2015 01:11:51 GMT
Ross you still having this issue?

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: Ross Laidlaw <rlaidlaw.open@gmail.com>
Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Date: Wednesday, May 13, 2015 at 2:37 AM
To: "dev@oodt.apache.org" <dev@oodt.apache.org>
Subject: Re: Request for advice on workflows

>Hi Chris,
>
>As a first step I got an XMLWorkflowRepository workflow running.  I then
>tried using the PackagedWorkflowRepository to create a workflow with the
>same tasks.  I got a sequential workflow up and running, similar to the
>following:
>
><cas:workflows ...>
>  <sequential id="urn:test:MainWorkflow" name="MainWorkflow">
>    <task id-ref="urn:test:TaskA"/>
>    <task id-ref="urn:test:TaskB"/>
>    <task id-ref="urn:test:TaskC"/>
>    <task id-ref="urn:test:TaskD"/>
>    <task id-ref="urn:test:TaskE"/>
>    <task id-ref="urn:test:TaskF"/>
>    <task id-ref="urn:test:TaskG"/>
>  </sequential>
>
>  <!-- task definitions -->
>  <task>...</task>
>
></cas:workflows>
>
>
>I found that I could run the workflow using ./wmgr-client --url
>[WorkflowManagerUrl] --sendEvent --eventName [WorkflowId]
>e.g.  ./wmgr-client --url http://localhost:9001 --sendEvent --eventName
>urn:test:MainWorkflow
>
>
>After getting the sequential workflow running, I wanted to run tasks C and
>D in parallel so I then tried adding a <parallel> section as follows:
>
><cas:workflows ...>
>  <sequential id="urn:test:MainWorkflow" name="MainWorkflow">
>    <task id-ref="urn:test:TaskA"/>
>    <task id-ref="urn:test:TaskB"/>
>    <parallel id="urn:test:ParallelWorkflow" name="ParallelWorkflow">
>      <task id-ref="urn:test:TaskC"/>
>      <task id-ref="urn:test:TaskD"/>
>    </parallel>
>    <task id-ref="urn:test:TaskE"/>
>    <task id-ref="urn:test:TaskF"/>
>    <task id-ref="urn:test:TaskG"/>
>  </sequential>
>  ...
></cas:workflows>
>
>
>The parallel section generates an additional task to be submitted called
>'Redirector Task' that wants to be executed in the 'high' queue using the
>BranchRedirector class:
>
>INFO: Task: [Redirector Task] has no required metadata fields
>May 13, 2015 1:27:29 AM
>org.apache.oodt.cas.resource.system.XmlRpcResourceManagerClient submitJob
>FINEST:
>[
>  {
>    job.queueName=high,
>    job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob,
>    job.name=redirector-322db9d2-4663-4b89-9af0-f8f97ad74c80,
>    job.id=, job.status=, job.load=2,
>    job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput
>  },
>  {
>
>task.instance.class=org.apache.oodt.cas.workflow.examples.BranchRedirector
>,
>    task.config={},
>    task.metadata={...}
>  }
>]
>
>
>I wasn't sure how to configure my setup to deal with this task.  From
>reading OODT-70 I gather that it might be trying to start up other
>wmgr-clients on batch stubs to run these tasks in separate workflows.  I
>tried setting up a local batch stub that accepts jobs from the 'high'
>queue.  When running the full workflow I get the following exception from
>the batch stub when it tries to execute the redirector task:
>
>INFO: Assigning job: [redirector-8561de94-2681-477b-8bcb-331d83224d5b] to
>node: [local-redirector]
>May 13, 2015 1:36:51 AM
>org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub
>genericExecuteJob
>INFO: stub attempting to execute class:
>[org.apache.oodt.cas.workflow.structs.TaskJob]
>org.apache.xmlrpc.XmlRpcClientException: Failure writing request
>  at
>org.apache.xmlrpc.XmlRpcClientRequestProcessor.encodeRequest(XmlRpcClientR
>equestProcessor.java:76)
>  at
>org.apache.xmlrpc.XmlRpcClientRequestProcessor.encodeRequestBytes(XmlRpcCl
>ientRequestProcessor.java:102)
>  at
>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:70)
>  at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>  at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>  at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>  at
>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.sendEvent(
>XmlRpcWorkflowManagerClient.java:477)
>  at
>org.apache.oodt.cas.workflow.examples.BranchRedirector.run(BranchRedirecto
>r.java:62)
>  at 
>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>  at
>org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run
>(XmlRpcBatchStub.java:229)
>  at java.lang.Thread.run(Thread.java:745)
>Caused by: org.apache.xmlrpc.XmlRpcException: null values not supported by
>XML-RPC
>  at org.apache.xmlrpc.XmlWriter.writeObject(XmlWriter.java:231)
>  at
>org.apache.xmlrpc.XmlRpcClientRequestProcessor.encodeRequest(XmlRpcClientR
>equestProcessor.java:72)
>  ... 10 more
>org.apache.oodt.cas.resource.structs.exceptions.JobInputException: Failed
>to run task
>  at 
>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:104)
>  at
>org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run
>(XmlRpcBatchStub.java:229)
>  at java.lang.Thread.run(Thread.java:745)
>Caused by:
>org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExcept
>ion:
>Failure writing request
>  at
>org.apache.oodt.cas.workflow.examples.BranchRedirector.run(BranchRedirecto
>r.java:64)
>  at 
>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>  ... 2 more
>May 13, 2015 1:36:51 AM
>org.apache.oodt.cas.resource.batchmgr.XmlRpcBatchMgrProxy run
>SEVERE: Job execution failed for jobId '2015-05-13T01:36:38.746-07:00' :
>batchstub.executeJob returned false
>java.lang.Exception: batchstub.executeJob returned false
>  at
>org.apache.oodt.cas.resource.batchmgr.XmlRpcBatchMgrProxy.run(XmlRpcBatchM
>grProxy.java:125)
>
>
>It looks like my sequential and parallel workflows have the required names
>and IDs.  Do you know what could be going wrong?  Maybe I'm missing a few
>vital configuration steps.
>
>Many thanks for your help,
>
>
>Ross
>
>
>
>
>
>
>
>On 8 May 2015 at 19:31, Ross Laidlaw <rlaidlaw.open@gmail.com> wrote:
>
>> Hi Chris,
>>
>> Many thanks for your advice.  I'll check out the
>> PackagedWorkflowRepository and will let you know how it goes.
>>
>> Ross
>>
>>
>>
>> > On May 8, 2015, at 12:58 PM, Mattmann, Chris A (3980) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> > Hey Ross:
>> >
>> > Great work! Try the PackagedWorkflowRepository. The best documentation
>> > that exists of it is in OODT-70 [1].
>> >
>> > I would use that to support the type of workflows that you’re doing.
>> > If this works we should make it the default since it’s supposed to
>> > be back compat with the XMLWorkflowRepositoryManager.
>> >
>> > Cheers,
>> > Chris
>> >
>> > P.S. see some of the ASO scripts I wrote to check on specific task/
>> > PGE status, etc.
>> >
>> >
>> > [1]https://issues.apache.org/jira/browse/OODT-70
>> >
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Chris Mattmann, Ph.D.
>> > Chief Architect
>> > Instrument Software and Science Data Systems Section (398)
>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > Office: 168-519, Mailstop: 168-527
>> > Email: chris.a.mattmann@nasa.gov
>> > WWW:  http://sunset.usc.edu/~mattmann/
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Adjunct Associate Professor, Computer Science Department
>> > University of Southern California, Los Angeles, CA 90089 USA
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >
>> >
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Ross Laidlaw <rlaidlaw.open@gmail.com>
>> > Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>> > Date: Friday, May 8, 2015 at 8:14 AM
>> > To: "dev@oodt.apache.org" <dev@oodt.apache.org>
>> > Subject: Request for advice on workflows
>> >
>> >> Hi All,
>> >>
>> >> I'm working on an OODT setup (v0.8.1) with several workflow tasks
>> defined
>> >> and working well individually.  I can use the wmgr-client to fire off
>> jobs
>> >> separately and all appears to be working as planned.  I'd like to
>> combine
>> >> these tasks into a workflow so that I don't have to run each set of
>> jobs.
>> >> For example, in the workflow I'd like to run several 'task A' jobs
>>and
>> >> when
>> >> all these have finished I'd like the workflow to automatically start
>> >> several 'task B' jobs, etc.
>> >>
>> >> I found some example workflows in the distribution, plus several
>>guides
>> up
>> >> on the wiki.  Is there a current recommended way of setting up a
>>simple
>> >> workflow?  Should I use workflow v2 or is it best to stick with v1?
>> Once
>> >> I've set up a workflow, how do I start it off?  It looks like it
>>might
>> be
>> >> possible to nest workflows, is that functionality available yet?
>> >>
>> >> Additionally, what's the best way to check on the status of currently
>> >> running jobs?  I've used the pcs_stat tool a lot and it's great for
>> >> checking on the overall status, but is there a recommended way to
>>check
>> >> e.g. if any 'task A' jobs are still running for a certain workflow
>>and
>> >> what
>> >> state they're currently in?
>> >>
>> >> Many thanks in advance for your advice,
>> >>
>> >>
>> >> Ross
>> >
>>

Mime
View raw message