Mailing-List: contact user-help@oodt.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@oodt.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "user@oodt.apache.org" <user@oodt.apache.org>
Date: Wed, 7 Mar 2012 22:00:49 -0800
Subject: Re: Workflow Question
Thread-Topic: Workflow Question
Thread-Index: Acz88NAJXOMb/HCsR3ikcxX7+UdDPg==
Message-ID: <E7975A57-A73D-4D38-9C5A-3E110CCDA421@jpl.nasa.gov>
References: <CB7A150E.B3BC%michael.cayanan@jpl.nasa.gov>
 <4F563BD7.6010401@nrao.edu>
 <CAOG8ptUnFGzDYfDt8PaZoc5TArowTBccnPv3MU4WAknk2xLVEg@mail.gmail.com>
 <FF5B9D1D-1CEC-406C-B6B6-69FD19504BC1@jpl.nasa.gov>
 <CAOG8ptW93nrTkBwHzatxyw92hFZWjcSnOQLMBykNtoqbdSeDyA@mail.gmail.com>
In-Reply-To: 
 <CAOG8ptW93nrTkBwHzatxyw92hFZWjcSnOQLMBykNtoqbdSeDyA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hey Bruce,

On Mar 7, 2012, at 5:33 AM, Bruce Barkstrom wrote:

> This situation appeared in my previous work on an
> archive design at LaRC.  The first instance was one
> where I wanted a reasonably secure file transfer - and
> it didn't look like there was an easy way to determine
> if a file transfer had completed so you could start
> checking on the file size.  The second instance had
> a Java Messenger Service function that would stash
> the message someplace until there was a request
> to pick it up.  The designer apparently couldn't see
> what would happen if the intended recipient object
> disappeared before receiving the message.  [This might
> be an interesting failure mode as the system storage
> starts to fill up with undeliverable messages.]

Two very applicable use cases where this type of=20
"freeing" of blocked resources comes in handy, that's
for sure.

Cheers,
Chris

>=20
> On Wed, Mar 7, 2012 at 1:26 AM, Mattmann, Chris A (388J)
> <chris.a.mattmann@jpl.nasa.gov> wrote:
>> Hey Bruce,
>>=20
>> On Mar 6, 2012, at 9:29 AM, Bruce Barkstrom wrote:
>>=20
>>> This question is a bit like asking the classic ftp software
>>> when has the file finished transferring.  The answer seems to me
>>> to be similar to the transaction protocols that produce
>>> reliable ACID behavior.  Perhaps it could be one of those
>>> cases that could be handled by carefully crafted exception
>>> handlers: system timed out after [limit], transaction rolled
>>> back with message to log, sender, and receiver.
>>=20
>> Agree! In pre 0.4 versions of OODT, this was supported by the
>> sense that you could always use the "resumeWorkflowInst" feature
>> to "unpause" a workflow waiting in a condition forever.
>>=20
>> In 0.4-SNAPSHOT (and the eventual release), workflow conditions now
>> support configurable timeouts, see OODT-215 [1] in general and more
>> specifically OODT-
>>=20
>> Cheers,
>> Chris
>>=20
>> [1] https://issues.apache.org/jira/browse/OODT-215
>> [2] https://issues.apache.org/jira/browse/OODT-207
>>=20
>>>=20
>>> Bruce B.
>>>=20
>>> On Tue, Mar 6, 2012 at 11:31 AM, Keith Cummings <kcumming@nrao.edu> wro=
te:
>>>> I have the same question Michael is asking (#1 below).  If a precondit=
ion
>>>> never passes, I do not want that workflow-task to indefinitely poll on=
 that
>>>> precondition.  At some point (preferably a configurable amount of time=
) it
>>>> should give up and fail (halt) the workflow.  Or is there another way =
to
>>>> address this Use Case?
>>>> -Keith Cummings
>>>>=20
>>>>=20
>>>> Cayanan, Michael D (388J) wrote:
>>>>=20
>>>> Hi Chris,
>>>>=20
>>>> On 3/2/12 9:36 PM, "Mattmann, Chris A (388J)"
>>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>=20
>>>>=20
>>>>=20
>>>> Hi Mike,
>>>>=20
>>>> On Mar 2, 2012, at 9:37 AM, Cayanan, Michael D (388J) wrote:
>>>>=20
>>>>=20
>>>>=20
>>>> Hi all,
>>>>=20
>>>> In the workflow.properties file, I see that I can configure the pollin=
g
>>>> time if a pre condition fails via the following property:
>>>>=20
>>>> org.apache.oodt.cas.workflow.engine.preConditionWaitTime
>>>>=20
>>>> I have a few questions:
>>>>=20
>>>> 1) Is this polling infinite?
>>>>=20
>>>>=20
>>>> What do you mean by "infinite"?
>>>>=20
>>>>=20
>>>>=20
>>>> If so, would you have to shut the workflow server down in order to sto=
p
>>>> this infinite polling? Or can you shut it down via the workflow manage=
r
>>>> client tool?
>>>>=20
>>>>=20
>>>> I'm not sure I follow you? The workflow manager's
>>>> ThreadPoolWorkflowEngine needs to evaluate preconditions, on a per
>>>> workflow-instance basis.
>>>> The only way to shut that down, is to shut down the workflow engine, a=
nd
>>>> thus implicitly the WM.
>>>>=20
>>>>=20
>>>> In other words, if the pre condition continually fails, will the workf=
low
>>>> ever timeout where the engine shuts down automatically?
>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>> 2) Is there a way to configure the pre condition wait time through the
>>>> PGE config file? For example, if I have 2 PGEs where I want each PGE t=
o
>>>> have different pre condition wait times, is this possible?
>>>>=20
>>>>=20
>>>> Can you tell me a specific use case or reason that you would want to h=
ave
>>>> different condition check times per PGE?
>>>>=20
>>>> In short, this can be supported by extending the
>>>> ThreadPoolWorkflowEngine, and then overriding or mapping the condition
>>>> wait
>>>> time to whatever is configured for the PGE. I'd severely caution again=
st
>>>> doing so, b/c you're really giving the developer or
>>>> workflow person an overt amount of control over an aspect of the syste=
m
>>>> that will affect (perceived) performance.
>>>>=20
>>>>=20
>>>> This was more of a curiosity question than anything. Personally, I don=
't
>>>> see the need to have a configurable pre condition wait time. In SMAP
>>>> world, there will be a case where a workflow requires two inputs, wher=
e
>>>> the inputs are coming from the outputs of 2 different workflows.
>>>> Furthermore, 1 of these inputs isn't expected to arrive like 10 hours
>>>> later after the first input arrives. The assumption here is that 1 inp=
ut
>>>> will trigger the workflow and thus the pre condition would be to check=
 for
>>>> the existence for that other input. So, if you have your precondition =
wait
>>>> time to some small interval like 30 seconds, would system performance =
be
>>>> affected in this case knowing that the precondition isn't expected to =
be
>>>> satisfied until 10 hours later?
>>>>=20
>>>> Hope this makes sense.
>>>>=20
>>>> Thanks again for your help!
>>>>=20
>>>> -Mike
>>>>=20
>>>>=20
>>>>=20
>>>> Cheers,
>>>> Chris
>>>>=20
>>>>=20
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Senior Computer Scientist
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 171-266B, Mailstop: 171-246
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Assistant Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>=20
>>=20
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>=20


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++