airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanchit Aggarwal <sanchitagarwal...@gmail.com>
Subject Re: Airavata GSoC 2013 Master Project
Date Sat, 27 Apr 2013 20:34:09 GMT
Hi Suresh,

I think the split of master project is perfect.

Adding to the project for workflow execution interface, we have to
include a functionality for validation of various input/output for
workflows execution.

Also if possible we can have the whole workflow to be executed in
stages, stages to be added by user. What I meant by stages is that
lets say we divide the whole workflow in three stage A-->B-->C, and
these stages will work as the checkpoints , if any of the stage get
failed we can save all the processing done till that stage and send
notification via monitoring tool so that later user can start the
workflow from that particular failed stage. please correct me if I am
wrong.

Also as suggested by Vijayedra, we have to see for the functionality
by which user can select specific variable /data of the workflow for
notification and tracking ,allowing user for custom monitoring.
Regards
Sanchit Aggarwal
MS Research (Computer Science)
Center for Visual Information Technology
IIIT Hyderabad, Gachibowli
Contact - 9581417330



On Sun, Apr 28, 2013 at 12:34 AM, Andun Sameera <andunslg@gmail.com> wrote:
> Hi Suresh,
>
> I also agree with your project decomposition. I have following comments,
>
> On Sun, Apr 28, 2013 at 12:06 AM, Suresh Marru <smarru@apache.org> wrote:
>
>> Hi Guys,
>>
>> Does any one have any proposal on how to break the project into smaller
>> gsoc projects?
>>
>> I think you all should work in parallel and as a team. There may be
>> projects two of you can implement the same idea. Example, the workflow
>> composer can be implemented in parallel by two students using different
>> libraries. But we need to make sure the JSON schema and workflow model
>> should be identical by both of them and so on.
>>
>> Here is one idea for the split (don't take it literally and propose other
>> ideas as well):
>>
>> * A project fully focused on Airavata data models for application and
>> workflows. So this person can work with rest of team to ensure a common
>> agreed upon JSON schema is drafted and works on the conversion as needed
>> with existing XML Schema.
>> * A project focused on the Airavata API and supporting the necessary
>> functionality needed to construct the web based interfaces. This project
>> basically have to understand the current Airavata Client API and existing
>> service interfaces of internal components (the REST and SOAP interfaces of
>> registry, interpreter, and messaging system -- GFac is invoked by
>> interpreter so it is not exposed outside).
>> * 2 projects on workflow composition interfaces themselves. These projects
>> have to interact with Airavata REST Service, parse the JSON messages,
>> construct the graph, generate the workflow xml (XWF) at the client or a
>> intermediary service.
>>
>
> There have to be a place where Application Developers can register there
> applications. I think these two project have to focus about that
> implementation also.
>
> * A project on workflow execution interfaces. This project has to
>> understand the current workflow inputs and workflow execution context and
>> workflow representation.
>> * A project on workflow monitoring - this project has to understand the
>> workflow tracking schema and ws messenger services.
>>
>> I am not sure if this is a good way to split the projects, but I am just
>> throwing an idea. Please don't look at them as separate projects. As we
>> have been discussing right from the beginning, these interleaved tasks
>> requires every one has to work on everything.
>>
>> Cheers,
>> Suresh
>>
>> On Apr 27, 2013, at 1:49 PM, Andun Sameera <andunslg@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I will bring some comments which I made earlier and which will help in
>> the
>> > discussion.
>> >
>> > On Fri, Apr 26, 2013 at 2:09 AM, Vijayendra Grampurohit <
>> > vijayendra.sdm@gmail.com> wrote:
>> >
>> >> Hi
>> >>
>> >> 1] Currently the workflow description has the graph representation form
>> >> xbaya .
>> >> The data for the Workflow Inputs come from the Registry .This data is in
>> >> wsdl form which is stored
>> >> in the Registry.
>> >> There is also some metadata associated with wsdl . So that the client
>> >> which has to bind the input values can use it.
>> >>
>> >> As we are discussing in the line of developing a  browser based
>> >> application.
>> >> For constructing workflow we can make use of below graph library's
>> >> http://jsplumbtoolkit.com/jquery/demo.html
>> >> http://raphaeljs.com/
>> >> In our project( Openshift workflows) we had used jsplumb library
>> >> which worked well with Angularjs .
>> >>
>> >> There are many more graph libraries which also can be explored.
>> >>
>> >
>> > I have digged deep in to the source code via debugging to see how the
>> XBaya
>> > create Workflows, persists them local file system, persists them in
>> > registry. I understood following things,
>> >
>> > - All the workflow design happens on top of the
>> > org.apache.airavata.xbaya.ui.graph.GraphCanvas
>> > - The workflow data represented via the
>> > org.apache.airavata.workflow.model.wf.Workflow ( I think this is the DAG
>> > representation which you are talking about in Airavata Workflow
>> > Architecture in [1])
>> > - Local file system persistence, retrieving done via
>> > org.apache.airavata.xbaya.core.generators.WorkflowFiler
>> > - Registry based persistence, retrieving has done via the REST client
>> > org.apache.airavata.rest.clientUserWorkflowResourceClient
>> >
>> > So If I focus on the task of implementing Web based Workflow Composer for
>> > Airavata, IMO we can reuse all three other than 1st one. 1st one have to
>> be
>> > replace with a JavaScript based implementation. When that handles design
>> > part all the other things related to workflow can be done reusing the
>> > others. If we use these or not, it is really valuable to understand the
>> > existing way of doing the thing.
>> >
>> > I also looked at rapheljs and jsplumb earlier. I think they suites the
>> > requirement and they comes under MIT license.
>> >
>> >
>> >> 2] Currently the data that is stored in Registry is of the form wsdl.
>> >> The workflow will use the data stored in the Registry .
>> >> For the new workflow as there is a discussion going on what kind of data
>> >> are we
>> >> going to store in registry   (i.e wsdl or JSON ). This can be worked
>> >> accordingly.
>> >>
>> >> 3] Monitoring tool : I am thinking in terms of a
>> >> browser plugin or a simple java script based web based monitoring which
>> >> will
>> >> notify users on workflow progress in real time.
>> >> This can be developed as a separate module .
>> >> The Monitoring tool subscribes to a pre-specified topic to which
>> >> Workflow Engine and GFac are publishing status notifications.Then the
>> >> monitoring
>> >> tool translates these messages and shows to to the user in the front
>> end.
>> >> Please see the image inserted.
>> >>
>> >> [image: Inline image 1]
>> >>
>> >> What we can also do , When user is executing a very large workflow
>> >> with hundreds of variables and doesn't want to track every thing, Then
>>  we
>> >> can have a option to
>> >> which allows customization in the monitoring tool .
>> >> My point is  we can have  features in which user can monitor
>> >> the variables or data he wants.
>> >>
>> >> Please correct me if I am wrong any where.
>> >>
>> >> Regards
>> >> Vijayendra
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Thu, Apr 25, 2013 at 7:08 PM, Suresh Marru <smarru@apache.org>
>> wrote:
>> >>
>> >>> On Apr 23, 2013, at 8:01 AM, Suresh Marru <smarru@apache.org>
wrote:
>> >>>
>> >>>> Great discussion Shameera & Subho.
>> >>>>
>> >>>> Looks like you guys have a fair idea on what needs to be done, as
you
>> >>> both state, a week or two into GSOC, you can narrow down on design and
>> >>> specifications and so forth. Given there are multiple students tackling
>> >>> these issues collectively, if there are hard decisions to make, I would
>> >>> suggest to try two approaches in parallel and quickly pick one after
a
>> >>> proof of concept.
>> >>>
>> >>> Can you all discuss the workflow composition strategies related to the
>> >>> data model the same way we discussed application descriptions?
>> >>>
>> >>> We need to decompose the master project into smaller, well aligned
>> >>> projects this weekend. So please start posting ideas on how to
>> sub-divide
>> >>> the projects.
>> >>>
>> >>> Suresh
>> >>>
>> >>>>
>> >>>> Suresh
>> >>>>
>> >>>> On Apr 23, 2013, at 3:43 AM, Subho Banerjee <subs.zero@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>>> Well exactly, as long as you can define standard way of
>> communication.
>> >>> That
>> >>>>> is, you can define in advance what should be a string, array
and what
>> >>>>> should be a integer etc. We have no problem.
>> >>>>>
>> >>>>> So, when you look at problems, with JSON <-> XML or the
other way
>> >>> round,
>> >>>>> they talk of the very general case (where you no nothing about
the
>> >>> data you
>> >>>>> are converting other than it is valid XML/JSON). There are a
myriad
>> of
>> >>>>> problems in that case, which you pointed out.
>> >>>>>
>> >>>>> But when there is standard, there is only one way of doing things,
>> and
>> >>> not
>> >>>>> several. I think that is the way forward. So what I am proposing
is
>> >>> maybe
>> >>>>> we all discuss and define this standard within the first week
of GSoC
>> >>>>> starting and then actually move into coding. So as long as we
work
>> >>> with the
>> >>>>> presumption that this will be done, we really dont have to worry
a
>> lot
>> >>>>> about this.
>> >>>>>
>> >>>>> Cheers,
>> >>>>> Subho.
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Apr 23, 2013 at 11:52 AM, Shameera Rathnayaka <
>> >>>>> shameerainfo@gmail.com> wrote:
>> >>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> On Tue, Apr 23, 2013 at 2:25 AM, Subho Banerjee <
>> subs.zero@gmail.com>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Some of these problems are very specific to what the
XML is
>> >>>>>> representing,
>> >>>>>>> it might not be an actual problem in Airavata,
>> >>>>>>> maybe some one more experienced with the codebase can
point this
>> out.
>> >>>>>>>
>> >>>>>>
>> >>>>>> All issues pointed out in the paper is not directly valid
to our
>> >>>>>> conversion, I didn't list the issues actually need to address
in
>> this
>> >>> case
>> >>>>>> because thought it is worth to read that introduction part
which
>> >>> explain
>> >>>>>> the all the issues we have with this conversion and give
us a solid
>> >>>>>> background of that.
>> >>>>>>
>> >>>>>>> 1. Anonymous values, Arrays, Implicit Typing, Character
sets -- I
>> >>>>>> really
>> >>>>>>> dont see these as problems, as long as you can agree
that all parts
>> >>> of
>> >>>>>>> airavata will treat the JSON in a standard (probably
we have to
>> >>> define
>> >>>>>>> this) way.
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> The issue with JSON array only comes when we try to convert
XML to
>> >>> JSON not
>> >>>>>> the other way. If we map with JSON, inputparameters and
>> >>> outputparameters in
>> >>>>>> the ServiceDescription.xsd will map with JSON Arrays. Therefore
we
>> >>> need to
>> >>>>>> solve this issue.
>> >>>>>>
>> >>>>>> JSON XML JSON
>> >>>>>> {"inputs":["test"]} --> <inputs>test<inputs>
 -->
>> {"inputs":["test"]}
>> >>>  //
>> >>>>>> correct one
>> >>>>>>                         --> {"inputs":"test"}     //
incorrect one
>> >>>>>>
>> >>>>>> 2. Namespaces, Processing Instructions -- Is this required?
>> >>>>>>
>> >>>>>>> Are separate namespaces used in Airavata? Only place
I can see this
>> >>>>>>> being
>> >>>>>>> used is probably in the WSDL, but if we can agree on
another way
>> >>>>>>> of communicating registered applications' I/O parameters
to the
>> >>> front
>> >>>>>>> end
>> >>>>>>> (JSON based), then maybe we can work around this (minor)
problem.
>> >>> Are
>> >>>>>>> custom processing instructions to the Xbaya XML parse
even used?
>> >>>>>>> 3. Attributes -- Again, this can be fixed easily
>> >>>>>>>
>> >>>>>>
>> >>>>>> Yes,attributes convertion will not be a big issues we can
solve it.
>> As
>> >>>>>> Lahiru mentioned in Hangout session namesapce handling is
not a big
>> >>> issue
>> >>>>>> with Airavata.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>> <array name="abc">
>> >>>>>>>   <element>1</element>
>> >>>>>>>   <element>2</element>
>> >>>>>>>   <element>3</element>
>> >>>>>>>   <element>4</element>
>> >>>>>>> </array>
>> >>>>>>>
>> >>>>>>> Can become
>> >>>>>>>
>> >>>>>>> {
>> >>>>>>>
>> >>>>>>>  abc : ['1', '2', '3', '4']
>> >>>>>>>
>> >>>>>>> }
>> >>>>>>>
>> >>>>>>
>> >>>>>> With this example it show us we need to change the XML message
>> format
>> >>> of
>> >>>>>> server side, which require to change the all schemas, If
we are
>> going
>> >>> to
>> >>>>>> change the schemas then we need to change the way it process
it in
>> >>> Ariavara
>> >>>>>> core. We have dropped our initial major requirement, which
is keep
>> the
>> >>>>>> Airavata Server side as it is.
>> >>>>>>
>> >>>>>> with this conversion we only deal with json strings, yes
we can send
>> >>> JSON
>> >>>>>> request with other formats supported by JSON like boolen,
null,
>> Number
>> >>>>>> etc.. But there is no way to get the same JSON from XML
as XML only
>> >>> deal
>> >>>>>> only with Strings. I think it is good if we can consume
a this
>> >>> features
>> >>>>>> with JSON.
>> >>>>>>
>> >>>>>> let say i need to send a integer or float to the server
using JSON
>> >>> then
>> >>>>>> proper way is to send {"<name>":123.45} this will
works fine but
>> >>> problem is
>> >>>>>> how we get the same output ?
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Shameera.
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Cheers,
>> >>>>>>> Subho.
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Best Regards,
>> >>>>>> Shameera Rathnayaka.
>> >>>>>>
>> >>>>>> Blog : http://shameerarathnayaka.blogspot.com/
>> >>>>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>> > Thanks!
>> >
>> > [1] - http://people.apache.org/~smarru/papers/airavata-gce11.pdf
>> > --
>> > Regards
>> > Andun S.L. Gunawardana
>> > Undergraduate
>> > Department of Computer Science And Engineering
>> > University of Moratuwa
>> > Sri Lanka
>> >
>> > Blog - http://www.insightforfuture.blogspot.com/
>> > LinkedIn - http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703
>> > Twitter -http://twitter.com/AndunSLG
>>
>>
> Thanks!
>
> --
> Regards
> Andun S.L. Gunawardana
> Undergraduate
> Department of Computer Science And Engineering
> University of Moratuwa
> Sri Lanka
>
> Blog - http://www.insightforfuture.blogspot.com/
> LinkedIn - http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703
> Twitter -http://twitter.com/AndunSLG

Mime
View raw message