oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Verma, Rishi (388J)" <Rishi.Ve...@jpl.nasa.gov>
Subject Re: Sharing workflow task metadata when using a resource manager
Date Wed, 26 Oct 2011 22:49:37 GMT
Thanks for the heads up Brian,

Actually, I am trying to pass workflow met from a WorkflowTaskInstance to a PGETaskInstance,
but I will keep this in mind for future cases!


On Oct 24, 2011, at 6:17 PM, Brian Foster wrote:

> if you are using CAS-PGE then in your pgeconfig.xml file you can specify on a any custom
metadata element the attribute workflowMet='true' and it will pass through to all the following
tasks in the workflow... for example: <metadata key='filename' val='data.dat' workflowMet='true'/>
> -Brian
> On Oct 24, 2011, at 5:45 PM, "Verma, Rishi (388J)" <Rishi.Verma@jpl.nasa.gov> wrote:
>> Hey Sheryl,
>> Thanks for the response!
>> Yeah, the workflow-manager log actually already spits out a lot of workflow task
instance info whenever it submits a task to a resource manager node.
>> It shows (for each task invoked):
>> * 'task.intance.class' name
>> * 'task.config' parameters
>> * 'task.metadata' contents
>> I did some tests, and concluded (from viewing task.metadata) that all task metadata
fed into the first task of my workflow (ie. when passing in met via: wmgr-client ... --key
name value...) showed up in the 'task.metadata' for all tasks in my workflow. However, if
my first task added new generated metadata, it would not show up in the 'task.metadata' of
subsequent tasks being called. The latter issue only occurs when using a resource manager
for task execution.
>> Thanks for the tip though, I will keep this in mind while trying out some new strategies.
>> rishi
>> On Oct 24, 2011, at 5:18 PM, Sheryl John wrote:
>>> Hi Rishi,
>>> Would getting workflowInstance metadata help you debug this problem ( using
>>> command-line option for the wmgr-client)? This will not give you the job
>>> metadata of course, but it gives out the job ids.
>>> And you could probably use the job id for checking out the metadata for
>>> that respective job. I'm not sure if there is code for this.
>>> On Mon, Oct 24, 2011 at 4:52 PM, Verma, Rishi (388J) <
>>> Rishi.Verma@jpl.nasa.gov> wrote:
>>>> Hi all,
>>>> I'm trying to pass generated metadata from a workflow task to another
>>>> workflow task (both in the same workflow) when using a resource manager to
>>>> execute workflow task jobs.
>>>> I've done this before successfully when workflow tasks within a given
>>>> workflow are run locally (by the workflow manager itself) but when I point
>>>> workflow manager to have tasks execute through a resource manager, my
>>>> generated metadata does not seem to transfer from one task to the next.
>>>> By "generated" metadata, I mean metadata that is added within the "run"
>>>> method of an implemented WorkflowTaskInstance. It's worth noting though,
>>>> that metadata passed into the initial XmlRpc call of a workflow task seems
>>>> to be passed to all subsequent tasks just fine. Just not generated metadata
>>>> - which passes only when not using a resource manager.
>>>> I'm trying to ascertain if this issue is a bug or not. To find out, could
>>>> someone elaborate a bit on which resmgr (or other) classes would include
>>>> code that actually shows metadata for a job being passed through a remotely
>>>> running job? I've been trying to find such code within the codebase but have
>>>> not come across it yet.
>>>> Thanks!
>>>> rishi
>>> -- 
>>> -Sheryl

View raw message