oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruce Barkstrom <brbarkst...@gmail.com>
Subject Re: Workflow Question
Date Thu, 08 Mar 2012 14:11:40 GMT
Quoting experiences may also be a simple sign of
getting older.

On Thu, Mar 8, 2012 at 1:00 AM, Mattmann, Chris A (388J)
<chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Bruce,
>
> On Mar 7, 2012, at 5:33 AM, Bruce Barkstrom wrote:
>
>> This situation appeared in my previous work on an
>> archive design at LaRC.  The first instance was one
>> where I wanted a reasonably secure file transfer - and
>> it didn't look like there was an easy way to determine
>> if a file transfer had completed so you could start
>> checking on the file size.  The second instance had
>> a Java Messenger Service function that would stash
>> the message someplace until there was a request
>> to pick it up.  The designer apparently couldn't see
>> what would happen if the intended recipient object
>> disappeared before receiving the message.  [This might
>> be an interesting failure mode as the system storage
>> starts to fill up with undeliverable messages.]
>
> Two very applicable use cases where this type of
> "freeing" of blocked resources comes in handy, that's
> for sure.
>
> Cheers,
> Chris
>
>>
>> On Wed, Mar 7, 2012 at 1:26 AM, Mattmann, Chris A (388J)
>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>> Hey Bruce,
>>>
>>> On Mar 6, 2012, at 9:29 AM, Bruce Barkstrom wrote:
>>>
>>>> This question is a bit like asking the classic ftp software
>>>> when has the file finished transferring.  The answer seems to me
>>>> to be similar to the transaction protocols that produce
>>>> reliable ACID behavior.  Perhaps it could be one of those
>>>> cases that could be handled by carefully crafted exception
>>>> handlers: system timed out after [limit], transaction rolled
>>>> back with message to log, sender, and receiver.
>>>
>>> Agree! In pre 0.4 versions of OODT, this was supported by the
>>> sense that you could always use the "resumeWorkflowInst" feature
>>> to "unpause" a workflow waiting in a condition forever.
>>>
>>> In 0.4-SNAPSHOT (and the eventual release), workflow conditions now
>>> support configurable timeouts, see OODT-215 [1] in general and more
>>> specifically OODT-
>>>
>>> Cheers,
>>> Chris
>>>
>>> [1] https://issues.apache.org/jira/browse/OODT-215
>>> [2] https://issues.apache.org/jira/browse/OODT-207
>>>
>>>>
>>>> Bruce B.
>>>>
>>>> On Tue, Mar 6, 2012 at 11:31 AM, Keith Cummings <kcumming@nrao.edu>
wrote:
>>>>> I have the same question Michael is asking (#1 below).  If a precondition
>>>>> never passes, I do not want that workflow-task to indefinitely poll on
that
>>>>> precondition.  At some point (preferably a configurable amount of time)
it
>>>>> should give up and fail (halt) the workflow.  Or is there another way
to
>>>>> address this Use Case?
>>>>> -Keith Cummings
>>>>>
>>>>>
>>>>> Cayanan, Michael D (388J) wrote:
>>>>>
>>>>> Hi Chris,
>>>>>
>>>>> On 3/2/12 9:36 PM, "Mattmann, Chris A (388J)"
>>>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi Mike,
>>>>>
>>>>> On Mar 2, 2012, at 9:37 AM, Cayanan, Michael D (388J) wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi all,
>>>>>
>>>>> In the workflow.properties file, I see that I can configure the polling
>>>>> time if a pre condition fails via the following property:
>>>>>
>>>>> org.apache.oodt.cas.workflow.engine.preConditionWaitTime
>>>>>
>>>>> I have a few questions:
>>>>>
>>>>> 1) Is this polling infinite?
>>>>>
>>>>>
>>>>> What do you mean by "infinite"?
>>>>>
>>>>>
>>>>>
>>>>> If so, would you have to shut the workflow server down in order to stop
>>>>> this infinite polling? Or can you shut it down via the workflow manager
>>>>> client tool?
>>>>>
>>>>>
>>>>> I'm not sure I follow you? The workflow manager's
>>>>> ThreadPoolWorkflowEngine needs to evaluate preconditions, on a per
>>>>> workflow-instance basis.
>>>>> The only way to shut that down, is to shut down the workflow engine,
and
>>>>> thus implicitly the WM.
>>>>>
>>>>>
>>>>> In other words, if the pre condition continually fails, will the workflow
>>>>> ever timeout where the engine shuts down automatically?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2) Is there a way to configure the pre condition wait time through the
>>>>> PGE config file? For example, if I have 2 PGEs where I want each PGE
to
>>>>> have different pre condition wait times, is this possible?
>>>>>
>>>>>
>>>>> Can you tell me a specific use case or reason that you would want to
have
>>>>> different condition check times per PGE?
>>>>>
>>>>> In short, this can be supported by extending the
>>>>> ThreadPoolWorkflowEngine, and then overriding or mapping the condition
>>>>> wait
>>>>> time to whatever is configured for the PGE. I'd severely caution against
>>>>> doing so, b/c you're really giving the developer or
>>>>> workflow person an overt amount of control over an aspect of the system
>>>>> that will affect (perceived) performance.
>>>>>
>>>>>
>>>>> This was more of a curiosity question than anything. Personally, I don't
>>>>> see the need to have a configurable pre condition wait time. In SMAP
>>>>> world, there will be a case where a workflow requires two inputs, where
>>>>> the inputs are coming from the outputs of 2 different workflows.
>>>>> Furthermore, 1 of these inputs isn't expected to arrive like 10 hours
>>>>> later after the first input arrives. The assumption here is that 1 input
>>>>> will trigger the workflow and thus the pre condition would be to check
for
>>>>> the existence for that other input. So, if you have your precondition
wait
>>>>> time to some small interval like 30 seconds, would system performance
be
>>>>> affected in this case knowing that the precondition isn't expected to
be
>>>>> satisfied until 10 hours later?
>>>>>
>>>>> Hope this makes sense.
>>>>>
>>>>> Thanks again for your help!
>>>>>
>>>>> -Mike
>>>>>
>>>>>
>>>>>
>>>>> Cheers,
>>>>> Chris
>>>>>
>>>>>
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Chris Mattmann, Ph.D.
>>>>> Senior Computer Scientist
>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> Office: 171-266B, Mailstop: 171-246
>>>>> Email: chris.a.mattmann@nasa.gov
>>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW:   http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>

Mime
View raw message