oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruce Barkstrom <brbarkst...@gmail.com>
Subject Re: Workflow Question
Date Wed, 07 Mar 2012 13:33:37 GMT
This situation appeared in my previous work on an
archive design at LaRC.  The first instance was one
where I wanted a reasonably secure file transfer - and
it didn't look like there was an easy way to determine
if a file transfer had completed so you could start
checking on the file size.  The second instance had
a Java Messenger Service function that would stash
the message someplace until there was a request
to pick it up.  The designer apparently couldn't see
what would happen if the intended recipient object
disappeared before receiving the message.  [This might
be an interesting failure mode as the system storage
starts to fill up with undeliverable messages.]

Bruce B.

On Wed, Mar 7, 2012 at 1:26 AM, Mattmann, Chris A (388J)
<chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Bruce,
>
> On Mar 6, 2012, at 9:29 AM, Bruce Barkstrom wrote:
>
>> This question is a bit like asking the classic ftp software
>> when has the file finished transferring.  The answer seems to me
>> to be similar to the transaction protocols that produce
>> reliable ACID behavior.  Perhaps it could be one of those
>> cases that could be handled by carefully crafted exception
>> handlers: system timed out after [limit], transaction rolled
>> back with message to log, sender, and receiver.
>
> Agree! In pre 0.4 versions of OODT, this was supported by the
> sense that you could always use the "resumeWorkflowInst" feature
> to "unpause" a workflow waiting in a condition forever.
>
> In 0.4-SNAPSHOT (and the eventual release), workflow conditions now
> support configurable timeouts, see OODT-215 [1] in general and more
> specifically OODT-
>
> Cheers,
> Chris
>
> [1] https://issues.apache.org/jira/browse/OODT-215
> [2] https://issues.apache.org/jira/browse/OODT-207
>
>>
>> Bruce B.
>>
>> On Tue, Mar 6, 2012 at 11:31 AM, Keith Cummings <kcumming@nrao.edu> wrote:
>>> I have the same question Michael is asking (#1 below).  If a precondition
>>> never passes, I do not want that workflow-task to indefinitely poll on that
>>> precondition.  At some point (preferably a configurable amount of time) it
>>> should give up and fail (halt) the workflow.  Or is there another way to
>>> address this Use Case?
>>> -Keith Cummings
>>>
>>>
>>> Cayanan, Michael D (388J) wrote:
>>>
>>> Hi Chris,
>>>
>>> On 3/2/12 9:36 PM, "Mattmann, Chris A (388J)"
>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>
>>>
>>>
>>> Hi Mike,
>>>
>>> On Mar 2, 2012, at 9:37 AM, Cayanan, Michael D (388J) wrote:
>>>
>>>
>>>
>>> Hi all,
>>>
>>> In the workflow.properties file, I see that I can configure the polling
>>> time if a pre condition fails via the following property:
>>>
>>> org.apache.oodt.cas.workflow.engine.preConditionWaitTime
>>>
>>> I have a few questions:
>>>
>>> 1) Is this polling infinite?
>>>
>>>
>>> What do you mean by "infinite"?
>>>
>>>
>>>
>>> If so, would you have to shut the workflow server down in order to stop
>>> this infinite polling? Or can you shut it down via the workflow manager
>>> client tool?
>>>
>>>
>>> I'm not sure I follow you? The workflow manager's
>>> ThreadPoolWorkflowEngine needs to evaluate preconditions, on a per
>>> workflow-instance basis.
>>> The only way to shut that down, is to shut down the workflow engine, and
>>> thus implicitly the WM.
>>>
>>>
>>> In other words, if the pre condition continually fails, will the workflow
>>> ever timeout where the engine shuts down automatically?
>>>
>>>
>>>
>>>
>>> 2) Is there a way to configure the pre condition wait time through the
>>> PGE config file? For example, if I have 2 PGEs where I want each PGE to
>>> have different pre condition wait times, is this possible?
>>>
>>>
>>> Can you tell me a specific use case or reason that you would want to have
>>> different condition check times per PGE?
>>>
>>> In short, this can be supported by extending the
>>> ThreadPoolWorkflowEngine, and then overriding or mapping the condition
>>> wait
>>> time to whatever is configured for the PGE. I'd severely caution against
>>> doing so, b/c you're really giving the developer or
>>> workflow person an overt amount of control over an aspect of the system
>>> that will affect (perceived) performance.
>>>
>>>
>>> This was more of a curiosity question than anything. Personally, I don't
>>> see the need to have a configurable pre condition wait time. In SMAP
>>> world, there will be a case where a workflow requires two inputs, where
>>> the inputs are coming from the outputs of 2 different workflows.
>>> Furthermore, 1 of these inputs isn't expected to arrive like 10 hours
>>> later after the first input arrives. The assumption here is that 1 input
>>> will trigger the workflow and thus the pre condition would be to check for
>>> the existence for that other input. So, if you have your precondition wait
>>> time to some small interval like 30 seconds, would system performance be
>>> affected in this case knowing that the precondition isn't expected to be
>>> satisfied until 10 hours later?
>>>
>>> Hope this makes sense.
>>>
>>> Thanks again for your help!
>>>
>>> -Mike
>>>
>>>
>>>
>>> Cheers,
>>> Chris
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW:   http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>>
>>>
>>>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>

Mime
View raw message