airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marlon Pierce <marpi...@iu.edu>
Subject Re: Workflow handling in the Application Catalog
Date Mon, 09 Jun 2014 20:00:22 GMT
Thanks, Eroma. For Step 1, we should also have a programmatic (API) way
to create workflows.  For questions 1 and 2, I think the answer is no to
both. Applications in the catalog are available individually and as part
of workflows.  #3 is a good point.  #4 is part of the larger "delete"
problem that we need to discuss.  I think for #5 we have one or the
other (and I personally think single applications are just special cases
of workflows).

Marlon

On 6/9/14 11:35 AM, Eroma Abeysinghe wrote:
> Hi Devs,
>
> I was thinking about workflows from system requirement point of view and
> IMO app catalog should cover below.
> (All given might be taken-care-of already; if so just ignore).
>
> When it comes to WFs (workflows) within App catalog we should be able to;
>
>    1. Create Workflows (More like adding the WF in to App catalog; as
>    actual WF creation is in XBAYA)
>    - When creating we should be able to give a name, description, ID
>       (generated) for the workflow. This would help with searing for
> WFs as well
>       as searching for experiments using workflow ID, name, etc....
>       - WFs can have a single node or multiple nodes. At least one node has
>       to have an application to execute. WFs can have one or many applications
>       - In WFs an output from one node can be a input to the next node
>    2. Publish & Un-publish Workflows
>       - We need to have the option of publishing and un-publishing them.
>       This way we can keep the WFs un-published till they are ready
> and then make
>       them available.
>    3. Edit Workflows
>       - We should be able to add remove nodes, applications from workflows.
>       4. Clone Workflows
>       - This is to assist with workflow creation. If we could clone it can
>       save time required to create WFs from scratch. Once cloned user should be
>       able to modify the new WF.
>       5. Import Workflows
>       - Importing from other gateways or an outside file. (Not so sure how
>       the importing works when it comes to XBAYA. If its imported from another
>       gateway the WF is already created using XBAYA. I guess then its matter of
>       having it inserted into catalog. How would the import work if
> the WF is not
>       even created in XBAYA)?
>       6. Search Workflows
>       - We should be able to search using
>          - Workflow name
>          - Workflow description
>          - Application (we give an application s the key and all workflows
>          using the given app will get listed)
>          - Searching for WFs giving experiment details such as exp name,
>          exp description, exp status
>          - Created user
>          - ... any more?
>          7. Delete Workflows
>       - This is to completely delete the workflow from app catalog. We may
>       not need this if we can un-publish
>
> Few questions
>
>    1. Can we have applications created only to be used in Workflows?
>    meaning we cannot use them individually in experiments.
>    2. Can we make an application available in workflows but unavailable for
>    individual use at a given time?
>    3. Currently when we search for experiments, we dont say whether  we are
>    looking for WFs or not. but in future this may be required
>    4. If we are deleting applications in the system if its is been used in
>    a WF are we going to allow deletion?
>    5. In an experiment can we use more than one WF? can we use WF and
>    application in an experiment?
>
> Thank You,
>
> Best Regards,
>
> Eroma
>
>
>
>
>
> On Mon, Jun 9, 2014 at 12:02 AM, Sachith Withana <swsachith@gmail.com>
> wrote:
>
>> Thanks Raman.
>>
>> You are exactly correct.
>>
>> I have three questions.
>>
>> 1. We should store the DAG only. But the problem is that DAG contains
>> which applications are used, and how they are connected.( which output goes
>> to which input). This is a lot of data that is already available in the
>> .wxf files that we use. So basically what you are proposing is shred the
>> .wxf files and store them in the App Catalog?
>>
>> 2. If I remember correctly, the workflows we used had web services as
>> nodes (SimpleMath workflow) , and they were not available in the registry
>> as applications before hand. How do you propose we handle that scenario?
>>
>> 3.  Overall, what's the purpose of having .wxf files ( WSDLS) as workflows
>> since now that we will only be using it to transfer data between the
>> client( XBaya) and the server correct?
>>
>>
>> On Thu, Jun 5, 2014 at 7:32 PM, Raminder Singh <raminderjsingh@gmail.com>
>> wrote:
>>
>>> Sachith,
>>>
>>> Workflow is a set of applications configured to work together. User need
>>> to define its individual applications and the next step is to create a
>>> workflow to configure the applications together. We should not save
>>> applications used in the workflow as part of workflow. String saved in
>>> workflow is a DAG (Directed acyclic graph) and it has reference to
>>> applications (currently services). Other details saved in the workflow are
>>> conditional blocks (if, for, while conditions), connections (input-output)
>>> and defaults inputs configured by the user. Doing this it will keep the
>>> structure of workflow DAG simple and application defined can be used in
>>> multiple workflows or run standalone if needed.  Please let me know if you
>>> have more questions
>>>
>>> Thanks
>>> Raminder
>>>
>>>
>>> On Jun 5, 2014, at 3:59 AM, Sachith Withana <swsachith@gmail.com> wrote:
>>>
>>> So what you are suggesting is instead of saving a one big string, we
>>> should shred it and store the nodes and their relationships separately so
>>> that it can be queriable?
>>>
>>>  But XBaya and the Workflow Interpreter would still be using the string
>>> right?
>>>
>>>
>>> On Tue, Jun 3, 2014 at 9:08 PM, Saminda Wijeratne <samindaw@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Sat, May 31, 2014 at 6:42 AM, Sachith Withana <swsachith@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, May 29, 2014 at 10:23 PM, Saminda Wijeratne <samindaw@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On Thu, May 29, 2014 at 1:15 AM, Sachith Withana <swsachith@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> When designing the Application Catalog for single Applications,
I ran
>>>>>>> into the issue of supporting workflows in the Application Catalog.
>>>>>>>
>>>>>>> Should we store the workflows ( workflow files) and keep ids,
which
>>>>>>> would be the handles for those workflows?
>>>>>>>
>>>>>> Thanks for bringing this up Sachith. In your opinion how do you think
>>>>>> the workflow should be represented through the CPI? Workflow string
or
>>>>>> Thrift object with workflow metadata distinguished or Thrift object
with
>>>>>> all workflow data distinguished or any other way? I suppose we need
to
>>>>>> figure-out what should be queried in a workflow.
>>>>>>
>>>>>
>>>>> For the functional purposes I think it would be better to store the
>>>>> workflow the way it is right now purely because we wouldn't want this
to
>>>>> affect the Workflow Interpreter behavior. But we can store meta-data
about
>>>>> the workflows to whatever the features we would want to support. Is there
a
>>>>> way to get the applications residing in a workflow without explicitly
>>>>> passing it to get the meta-data?
>>>>>
>>>> Workflow nodes specifically does directly correspond to applications but
>>>> service descriptors. The current workflow model object have functions to
>>>> return these service descriptors.
>>>>
>>>>
>>>>>
>>>>>>> We would have to keep track of the applications that are in that
>>>>>>> workflow so that the deployment related data would be reused.
>>>>>>>
>>>>>>> Similarly to the single Applications, the sharing and other features
>>>>>>> would be available for the workflows as well.
>>>>>>>
>>>>>>> Is there any more details that I should be concerned about when
>>>>>>> implementing the aforementioned approach?
>>>>>>>
>>>>>> IMO we can generalize workflows as a composite application with
>>>>>> special properties. For example a workflow could simply be another
>>>>>> deployment for an Application interface. wdyt?
>>>>>>
>>>>>
>>>>> I agree. I'll send you the current design I have which generalize this.
>>>>>
>>>>>
>>>>>>
>>>>>>> If we are planning to add searching capability with the workflows
as
>>>>>>> well, then we'd have to store the applications that are used
in the
>>>>>>> workflows, separately in the database as well instead of storing
it as a
>>>>>>> whole.
>>>>>>>
>>>>>> You mean node information here when you say "applications" right?
>>>>>>
>>>>> yep. wdyt?
>>>>>
>>>> I think it brings us back to whether want to save the workflow as a
>>>> single blob or as separate data for the graph. IMO what we send to the
>>>> registry should be a complex workflow object and we should be able to query
>>>> those objects from the registry based on their properties. How registry
>>>> saves this complex object (as a single blob, metadata+single blob or
>>>> metadata+graph data) is immaterial for the user of the registry/app catalog.
>>>>
>>>>
>>>>
>>>>>>> Any suggestions/comments on the matter is highly appreciated.
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>>  Sachith Withana
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>>  Sachith Withana
>>>>>
>>>>>
>>>
>>> --
>>> Thanks,
>>> Sachith Withana
>>>
>>>
>>>
>>
>> --
>> Thanks,
>>  Sachith Withana
>>
>>
>


Mime
View raw message