oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (388J)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: Workflow Question
Date Thu, 08 Mar 2012 06:00:49 GMT
Hey Bruce,

On Mar 7, 2012, at 5:33 AM, Bruce Barkstrom wrote:

> This situation appeared in my previous work on an
> archive design at LaRC.  The first instance was one
> where I wanted a reasonably secure file transfer - and
> it didn't look like there was an easy way to determine
> if a file transfer had completed so you could start
> checking on the file size.  The second instance had
> a Java Messenger Service function that would stash
> the message someplace until there was a request
> to pick it up.  The designer apparently couldn't see
> what would happen if the intended recipient object
> disappeared before receiving the message.  [This might
> be an interesting failure mode as the system storage
> starts to fill up with undeliverable messages.]

Two very applicable use cases where this type of 
"freeing" of blocked resources comes in handy, that's
for sure.

Cheers,
Chris

> 
> On Wed, Mar 7, 2012 at 1:26 AM, Mattmann, Chris A (388J)
> <chris.a.mattmann@jpl.nasa.gov> wrote:
>> Hey Bruce,
>> 
>> On Mar 6, 2012, at 9:29 AM, Bruce Barkstrom wrote:
>> 
>>> This question is a bit like asking the classic ftp software
>>> when has the file finished transferring.  The answer seems to me
>>> to be similar to the transaction protocols that produce
>>> reliable ACID behavior.  Perhaps it could be one of those
>>> cases that could be handled by carefully crafted exception
>>> handlers: system timed out after [limit], transaction rolled
>>> back with message to log, sender, and receiver.
>> 
>> Agree! In pre 0.4 versions of OODT, this was supported by the
>> sense that you could always use the "resumeWorkflowInst" feature
>> to "unpause" a workflow waiting in a condition forever.
>> 
>> In 0.4-SNAPSHOT (and the eventual release), workflow conditions now
>> support configurable timeouts, see OODT-215 [1] in general and more
>> specifically OODT-
>> 
>> Cheers,
>> Chris
>> 
>> [1] https://issues.apache.org/jira/browse/OODT-215
>> [2] https://issues.apache.org/jira/browse/OODT-207
>> 
>>> 
>>> Bruce B.
>>> 
>>> On Tue, Mar 6, 2012 at 11:31 AM, Keith Cummings <kcumming@nrao.edu> wrote:
>>>> I have the same question Michael is asking (#1 below).  If a precondition
>>>> never passes, I do not want that workflow-task to indefinitely poll on that
>>>> precondition.  At some point (preferably a configurable amount of time) it
>>>> should give up and fail (halt) the workflow.  Or is there another way to
>>>> address this Use Case?
>>>> -Keith Cummings
>>>> 
>>>> 
>>>> Cayanan, Michael D (388J) wrote:
>>>> 
>>>> Hi Chris,
>>>> 
>>>> On 3/2/12 9:36 PM, "Mattmann, Chris A (388J)"
>>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>> 
>>>> 
>>>> 
>>>> Hi Mike,
>>>> 
>>>> On Mar 2, 2012, at 9:37 AM, Cayanan, Michael D (388J) wrote:
>>>> 
>>>> 
>>>> 
>>>> Hi all,
>>>> 
>>>> In the workflow.properties file, I see that I can configure the polling
>>>> time if a pre condition fails via the following property:
>>>> 
>>>> org.apache.oodt.cas.workflow.engine.preConditionWaitTime
>>>> 
>>>> I have a few questions:
>>>> 
>>>> 1) Is this polling infinite?
>>>> 
>>>> 
>>>> What do you mean by "infinite"?
>>>> 
>>>> 
>>>> 
>>>> If so, would you have to shut the workflow server down in order to stop
>>>> this infinite polling? Or can you shut it down via the workflow manager
>>>> client tool?
>>>> 
>>>> 
>>>> I'm not sure I follow you? The workflow manager's
>>>> ThreadPoolWorkflowEngine needs to evaluate preconditions, on a per
>>>> workflow-instance basis.
>>>> The only way to shut that down, is to shut down the workflow engine, and
>>>> thus implicitly the WM.
>>>> 
>>>> 
>>>> In other words, if the pre condition continually fails, will the workflow
>>>> ever timeout where the engine shuts down automatically?
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 2) Is there a way to configure the pre condition wait time through the
>>>> PGE config file? For example, if I have 2 PGEs where I want each PGE to
>>>> have different pre condition wait times, is this possible?
>>>> 
>>>> 
>>>> Can you tell me a specific use case or reason that you would want to have
>>>> different condition check times per PGE?
>>>> 
>>>> In short, this can be supported by extending the
>>>> ThreadPoolWorkflowEngine, and then overriding or mapping the condition
>>>> wait
>>>> time to whatever is configured for the PGE. I'd severely caution against
>>>> doing so, b/c you're really giving the developer or
>>>> workflow person an overt amount of control over an aspect of the system
>>>> that will affect (perceived) performance.
>>>> 
>>>> 
>>>> This was more of a curiosity question than anything. Personally, I don't
>>>> see the need to have a configurable pre condition wait time. In SMAP
>>>> world, there will be a case where a workflow requires two inputs, where
>>>> the inputs are coming from the outputs of 2 different workflows.
>>>> Furthermore, 1 of these inputs isn't expected to arrive like 10 hours
>>>> later after the first input arrives. The assumption here is that 1 input
>>>> will trigger the workflow and thus the pre condition would be to check for
>>>> the existence for that other input. So, if you have your precondition wait
>>>> time to some small interval like 30 seconds, would system performance be
>>>> affected in this case knowing that the precondition isn't expected to be
>>>> satisfied until 10 hours later?
>>>> 
>>>> Hope this makes sense.
>>>> 
>>>> Thanks again for your help!
>>>> 
>>>> -Mike
>>>> 
>>>> 
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> 
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Senior Computer Scientist
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 171-266B, Mailstop: 171-246
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Assistant Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mime
View raw message