oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cayanan, Michael D (388J)" <michael.d.caya...@jpl.nasa.gov>
Subject Re: Workflow 2
Date Fri, 21 Dec 2012 08:30:18 GMT
Hey Chris,

My answers are below in blue:

From: <Mattmann>, Chris Mattmann <Chris.A.Mattmann@jpl.nasa.gov<mailto:Chris.A.Mattmann@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Date: Thursday, December 20, 2012 4:43 PM
To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Subject: Re: Workflow 2

Hey Mike,


From: <Cayanan>, "Michael D (388J)" <michael.d.cayanan@jpl.nasa.gov<mailto:michael.d.cayanan@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Date: Thursday, December 20, 2012 4:37 PM
To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Subject: Re: Workflow 2

Hey Chris,

Sorry, I tried attaching my entire workflow distribution package to the previous e-mail so
that you can try it out on your local machine, but it looks like the exchange server didn't
like that.

I'm attaching just the workflow.properties and the workflow 2 policy that I'm using.

Thanks no problem! Answers, read on below :)

I added a link from the front page under the 'Workflow/CAS-PGE/Resource Manager' area :D

Thanks!

I've been playing around with the latest Workflow from the trunk using the TestParallel example
given in the examples/wengine folder so that I can learn more about how to create Workflow
2 policy.

I ran into a few issues that I was hoping you could help me out with:

1) The timeout condition doesn't appear to be working. I changed the timeout to '600' seconds
instead of '20' to let the event run for a longer period of time. However, the event appears
to run the same amount of time with the timeout=20 vs timeout=600.

Yep this isn't support in the PrioritizedQueueBasedWorkflowEngine yet. Try using the ThreadPoolWorkflowEngine,
with the PackagedWorkflowRepository this should work.

I tried using the ThreadPoolWorkflowEngine instead and it still doesn't appear to be working.
Still runs In fact, it doesn't seem to recognize the conditions that I'm setting at all. I
modified my policy to where the 'HelloWorld' and 'GoodbyeWorld' task definitions each have
a 'FalseCondition' now. By doing this, this means that the tasks would never run, correct?
The TestParallel example is still executing the 'HelloWorld' and 'GoodByeWorld' tasks. I'm
attaching my modified workflow2 policy to this e-mail.

2) I'm trying to pass in a 'numSeconds' metadata to the workflow event so that the intensive
task can run for 60 seconds instead of the default 10 seconds. But the log keeps indicating
that the number of seconds is set at 10. This is how I'm triggering the event on the command-line:

% ./wmgr-client -u http://localhost:9001 -en urn:oodt:TestParallel --metaData --key numSeconds
60 -op -se

Dec 20, 2012 4:25:34 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
INFO: WorkflowManager: Received event: urn:oodt:TestParallel
Dec 20, 2012 4:25:34 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
INFO: WorkflowManager: Workflow HelloGoodbye retrieved for event urn:oodt:TestParallel
Dec 20, 2012 4:25:34 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent
INFO: WorkflowManager: Workflow Parallel Single Task Processor Intensive Task retrieved for
event urn:oodt:TestParallel
Dec 20, 2012 4:25:36 PM org.apache.oodt.cas.workflow.engine.TaskQuerier run
INFO: Added processor with priority: [MEDIUM : 5.0]
Task: Num seconds: 10
Dec 20, 2012 4:25:36 PM org.apache.oodt.cas.workflow.engine.TaskQuerier run
INFO: Added processor with priority: [MEDIUM : 5.0]
Hello World: Chris
Dec 20, 2012 4:25:36 PM org.apache.oodt.cas.workflow.engine.runner.AsynchronousLocalEngineRunner$1
run
INFO: Task: [Hello World] for instance id: [f06e0357-4b04-11e2-95a4-edbfb4ca9bb3] completed
successfully
Dec 20, 2012 4:25:36 PM org.apache.oodt.cas.workflow.engine.TaskQuerier run
INFO: Added processor with priority: [MEDIUM : 5.0]
Goodbye World: Chris
Dec 20, 2012 4:25:36 PM org.apache.oodt.cas.workflow.engine.runner.AsynchronousLocalEngineRunner$1
run
INFO: Task: [Goodbye World] for instance id: [f0738198-4b04-11e2-95a4-edbfb4ca9bb3] completed
successfully
Dec 20, 2012 4:25:46 PM org.apache.oodt.cas.workflow.engine.runner.AsynchronousLocalEngineRunner$1
run
INFO: Task: [Processor Intensive Task] for instance id: [f0632de6-4b04-11e2-95a4-edbfb4ca9bb3]
completed successfully

Yep it's not honoring these properties yet — but the ThreadPoolWorkflowEngine will. See
latest status in OODT-491 [1].

Okay, with the ThreadPoolWorkflowEngine, it is recognizing the numSeconds metadata that I'm
passing in.

3) When I try to look at the list of Workflow Instances that have ran or are running currently,
I get an exception thrown:

% ./wmgr-client -u http://localhost:9001 -op -winsts
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.workflow.structs.exceptions.EngineException:
Exception getting workflow instances from workflow engine: Message: null
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.getWorkflowInstances(XmlRpcWorkflowManagerClient.java:653)
at org.apache.oodt.cas.workflow.cli.action.GetWorkflowInstsCliAction.execute(GetWorkflowInstsCliAction.java:42)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.main(XmlRpcWorkflowManagerClient.java:678)
ERROR: Failed to get workflow instances from URL 'http://localhost:9001' : java.lang.Exception:
org.apache.oodt.cas.workflow.structs.exceptions.EngineException: Exception getting workflow
instances from workflow engine: Message: null

In the Workflow log, it shows the following exception:

Dec 20, 2012 4:26:52 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager getWorkflowInstances
INFO: Getting workflow instances: retrieved: 10 instances
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at org.apache.oodt.cas.workflow.util.XmlRpcStructFactory.getXmlRpcWorkflow(XmlRpcStructFactory.java:227)
at org.apache.oodt.cas.workflow.util.XmlRpcStructFactory.getXmlRpcWorkflowInstance(XmlRpcStructFactory.java:114)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager.getWorkflowInstances(XmlRpcWorkflowManager.java:448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
at java.lang.Thread.run(Thread.java:680)

I'm guessing I'm forgetting to set something in the policy. However, I'm not sure what is
missing.

Nah you're fine. I think this is an issue with the Serialization/Deserialization of ParentChildWorkflows.
Can you file a JIRA issue for this one and link it to OODT-491 [1]?

I filed a JIRA issue for this: https://issues.apache.org/jira/browse/OODT-549

A different way that works and that will give you the same result is to run:

./wmgr-client —url http://localhost:9001 —operation —getFirstPage

That worked for me. However, I found an interesting bug. Basically, once you restart the Workflow
Manager Server and then try to run that 'getFirstPage' command, it throws an exception. Below
shows the set of commands that you can run to reproduce the issue. Let me know if you think
this should be a JIRA issue as well. On a side note, when I stopped the Workflow Manager Server,
deleted the Lucene Workflow Instance Repository directory, then started the Worfklow Manager
again, the error goes away.

% ./wmgr-client -u http://localhost:9001 --operation --getFirstPage
Page: [num=0,pageSize=0,totalPages=0]

% ./wmgr-client -u http://localhost:9001 -en urn:oodt:TestParallel --metaData --key numSeconds
15 -op -se
Sending event 'urn:oodt:TestParallel'... SUCCESS

% ./wmgr-client -u http://localhost:9001 --operation --getFirstPage
Page: [num=1,pageSize=20,totalPages=1]
Instance: [id=4e5a2d0c-4b47-11e2-977b-c39e66f8f1d5, status=STARTED, currentTask=urn:oodt:IntensiveTask,
workflow=Parallel Single Task Processor Intensive Task, wallClockTime=0.09376666666666668,
currentTaskWallClockTime=0.0]
Instance: [id=4e4e6d3b-4b47-11e2-977b-c39e66f8f1d5, status=FINISHED, currentTask=urn:oodt:GoodbyeWorld,
workflow=HelloGoodbye, wallClockTime=0.0049, currentTaskWallClockTime=0.0]

% ./wmgr restart
-n Shutting down cas workflow manager:
OK
-n Starting cas workflow manager:
OK
Dec 21, 2012 12:21:50 AM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager loadProperties
INFO: Loading Workflow Manager Configuration Properties from: [../etc/workflow.properties]
Dec 21, 2012 12:21:50 AM org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactory
getResmgrUrl
INFO: No Resource Manager URL provided or malformed URL: executing jobs locally. URL: []
Adding condition: [Timeout Condition] to parent workflow: [TestParallel]
Dec 21, 2012 12:21:50 AM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init>
INFO: Workflow Manager started by mcayanan

% ./wmgr-client -u http://localhost:9001 --operation --getFirstPage
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: java.lang.NullPointerException
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.getFirstPage(XmlRpcWorkflowManagerClient.java:122)
at org.apache.oodt.cas.workflow.cli.action.GetFirstPageCliAction.execute(GetFirstPageCliAction.java:47)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.main(XmlRpcWorkflowManagerClient.java:678)
ERROR: Failed to get first page of workflows : java.lang.Exception: java.lang.NullPointerException

Cheers,
Chris

[1] http://isuses.apache.org/jira/browse/OODT-491

Cheers,
Mike


From: <Mattmann>, Chris Mattmann <Chris.A.Mattmann@jpl.nasa.gov<mailto:Chris.A.Mattmann@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Date: Wednesday, December 19, 2012 6:54 PM
To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Subject: Re: Workflow 2

Mike you ROCK! I'll help you expand this. Also I'll add a link from the front page of the
Wiki if you haven't alreadyŠ

Cheers,
Chris

From: <Cayanan>, "Michael D (388J)" <michael.d.cayanan@jpl.nasa.gov<mailto:michael.d.cayanan@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Date: Tuesday, December 18, 2012 11:10 AM
To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Subject: Re: Workflow 2

Hi all,

I've started a Quick Start Guide for Workflow2 on our wiki:

https://cwiki.apache.org/confluence/display/OODT/Workflow2+Quick+Start+Guide

It contains some workflow.properties configuration that's needed and some FAQ on Workflow2.

Feel free to add to this page for those that our Workflow2 gurus. :) I'm a newbie to Workflow2
so I intend on adding to it at some point in the near future once I stand up a Workflow using
Workflow2 policy.

Cheers,
Mike
From: <Cayanan>, "Michael D (388J)" <michael.d.cayanan@jpl.nasa.gov<mailto:michael.d.cayanan@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Date: Thursday, December 13, 2012 3:18 PM
To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Subject: Re: Workflow 2

Sounds good. Ya I saw Brian's Wengine User Guide on the wiki, but wasn't sure if it was outdated
or not. It does have a lot of useful information in there.

Good idea on wanting to start up a Workflow 2 Quick Start page. I will definitely start one
up and include information in this thread on there as a starting point and we'll build on
it as we go along.

Cheers,
Mike

From: <Mattmann>, Chris Mattmann <Chris.A.Mattmann@jpl.nasa.gov<mailto:Chris.A.Mattmann@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Date: Thursday, December 13, 2012 11:43 AM
To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Subject: Re: Workflow 2

Hi Mike,


From: <Cayanan>, "Michael D (388J)" <michael.d.cayanan@jpl.nasa.gov<mailto:michael.d.cayanan@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Date: Thursday, December 13, 2012 10:41 AM
To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Subject: Re: Workflow 2

Thanks for the helpful information Chris. This is a good starting point!

Woot! Can you help me start documenting this on the Apache OODT wiki? I threw up Brian's Wengine
user guide, which is useful in its own right since we still have the branch out there, and
some folks using it, but how about we start a new "Workflow 2 Quick Start" page on the wiki
and through this and stuff below on it?

If I wanted to specify parallel tasks in a workflow, that would mean I'd have to use workflow
2 policy instead of workflow 1 policy, correct? If this isn't true, how does one specify parallel
tasks using workflow 1 policy?

Yep you would use workflow2 style policy of which there are examples in src/main/resources/wengine
in the trunk.

Thanks!

Cheers,
Chris


Thanks,
Mike

From: <Mattmann>, Chris Mattmann <Chris.A.Mattmann@jpl.nasa.gov<mailto:Chris.A.Mattmann@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Date: Wednesday, December 12, 2012 6:50 PM
To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Subject: Re: Workflow 2

Hey Mike,


From: <Cayanan>, "Michael D (388J)" <michael.d.cayanan@jpl.nasa.gov<mailto:michael.d.cayanan@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Date: Wednesday, December 12, 2012 10:15 AM
To: "user@oodt.apache.org<mailto:user@oodt.apache.org>" <user@oodt.apache.org<mailto:user@oodt.apache.org>>
Subject: Workflow 2

Hi all,

I would like to start using the latest and greatest Workflow in the trunk. Has anyone been
using it lately?

That would be me :)

I'm trying to figure out which configuration files are needed now in
order to run this latest version. Or is it backwards compatible?

It's fully back compat. So feel free to point wengine at your existing CAS workflow 1 policy
and it should load fine.

 In 0.3, I know I had to configure a couple of xml files: events.xml, task.xml, conditions.xml.
However, in the examples directory, I do see a wengine folder, where it looks like those 3
config files aren't needed anymore?

They don't have to be used, but they can be used.

If someone can point me to some sample set of files I can use as a reference, that would be
awesome! :D

Sure, check this out:

http://svn.apache.org/repos/asf/oodt/trunk/workflow/src/main/resources/examples/

Those files should work fine. Give it a try. As for your workflow.properties file, it should
look like this:

# wengine properties
# define workflow prioritizer class to use for sorting workflow tasks
org.apache.oodt.cas.workflow.wengine.prioritizer=org.apache.oodt.cas.workflow.structs.FILOPrioritySorter
org.apache.oodt.cas.workflow.wengine.taskquerier.waitSeconds=2

# the maximum number of threads to be used by the asynchronous engine runner
org.apache.oodt.cas.workflow.engine.asynchronous.runner.num.threads=25

You'll need those properties at a minimum. Then, at the top of your file, select:

# workflow repository factory
workflow.repo.factory = org.apache.oodt.cas.workflow.repository.PackagedWorkflowRepositoryFactory

# workflow engine factory
workflow.engine.factory = org.apache.oodt.cas.workflow.engine.PrioritizedQueueBasedWorkflowEngineFactory

# engine runner factory
workflow.wengine.runner.factory=org.apache.oodt.cas.workflow.engine.runner.AsynchronousLocalEngineRunnerFactory

Remember for your packaged workflow repo to set:

# wengine-style packaged workflow repo properties
org.apache.oodt.cas.workflow.wengine.packagedRepo.dir.path = /path/to/wengine/workflow/files

And to define your wengine lifecycle:

org.apache.oodt.cas.workflow.lifecycle.filePath=/usr/local/workflow/policy/wengine/wengine-lifecycle.xml

HTH!

Cheers,
Chris


Mime
View raw message