airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andun Sameera <andun...@gmail.com>
Subject Re: Airavata GSoC 2013 Master Project
Date Sat, 27 Apr 2013 19:04:21 GMT
Hi Suresh,

I also agree with your project decomposition. I have following comments,

On Sun, Apr 28, 2013 at 12:06 AM, Suresh Marru <smarru@apache.org> wrote:

> Hi Guys,
>
> Does any one have any proposal on how to break the project into smaller
> gsoc projects?
>
> I think you all should work in parallel and as a team. There may be
> projects two of you can implement the same idea. Example, the workflow
> composer can be implemented in parallel by two students using different
> libraries. But we need to make sure the JSON schema and workflow model
> should be identical by both of them and so on.
>
> Here is one idea for the split (don't take it literally and propose other
> ideas as well):
>
> * A project fully focused on Airavata data models for application and
> workflows. So this person can work with rest of team to ensure a common
> agreed upon JSON schema is drafted and works on the conversion as needed
> with existing XML Schema.
> * A project focused on the Airavata API and supporting the necessary
> functionality needed to construct the web based interfaces. This project
> basically have to understand the current Airavata Client API and existing
> service interfaces of internal components (the REST and SOAP interfaces of
> registry, interpreter, and messaging system -- GFac is invoked by
> interpreter so it is not exposed outside).
> * 2 projects on workflow composition interfaces themselves. These projects
> have to interact with Airavata REST Service, parse the JSON messages,
> construct the graph, generate the workflow xml (XWF) at the client or a
> intermediary service.
>

There have to be a place where Application Developers can register there
applications. I think these two project have to focus about that
implementation also.

* A project on workflow execution interfaces. This project has to
> understand the current workflow inputs and workflow execution context and
> workflow representation.
> * A project on workflow monitoring - this project has to understand the
> workflow tracking schema and ws messenger services.
>
> I am not sure if this is a good way to split the projects, but I am just
> throwing an idea. Please don't look at them as separate projects. As we
> have been discussing right from the beginning, these interleaved tasks
> requires every one has to work on everything.
>
> Cheers,
> Suresh
>
> On Apr 27, 2013, at 1:49 PM, Andun Sameera <andunslg@gmail.com> wrote:
>
> > Hi,
> >
> > I will bring some comments which I made earlier and which will help in
> the
> > discussion.
> >
> > On Fri, Apr 26, 2013 at 2:09 AM, Vijayendra Grampurohit <
> > vijayendra.sdm@gmail.com> wrote:
> >
> >> Hi
> >>
> >> 1] Currently the workflow description has the graph representation form
> >> xbaya .
> >> The data for the Workflow Inputs come from the Registry .This data is in
> >> wsdl form which is stored
> >> in the Registry.
> >> There is also some metadata associated with wsdl . So that the client
> >> which has to bind the input values can use it.
> >>
> >> As we are discussing in the line of developing a  browser based
> >> application.
> >> For constructing workflow we can make use of below graph library's
> >> http://jsplumbtoolkit.com/jquery/demo.html
> >> http://raphaeljs.com/
> >> In our project( Openshift workflows) we had used jsplumb library
> >> which worked well with Angularjs .
> >>
> >> There are many more graph libraries which also can be explored.
> >>
> >
> > I have digged deep in to the source code via debugging to see how the
> XBaya
> > create Workflows, persists them local file system, persists them in
> > registry. I understood following things,
> >
> > - All the workflow design happens on top of the
> > org.apache.airavata.xbaya.ui.graph.GraphCanvas
> > - The workflow data represented via the
> > org.apache.airavata.workflow.model.wf.Workflow ( I think this is the DAG
> > representation which you are talking about in Airavata Workflow
> > Architecture in [1])
> > - Local file system persistence, retrieving done via
> > org.apache.airavata.xbaya.core.generators.WorkflowFiler
> > - Registry based persistence, retrieving has done via the REST client
> > org.apache.airavata.rest.clientUserWorkflowResourceClient
> >
> > So If I focus on the task of implementing Web based Workflow Composer for
> > Airavata, IMO we can reuse all three other than 1st one. 1st one have to
> be
> > replace with a JavaScript based implementation. When that handles design
> > part all the other things related to workflow can be done reusing the
> > others. If we use these or not, it is really valuable to understand the
> > existing way of doing the thing.
> >
> > I also looked at rapheljs and jsplumb earlier. I think they suites the
> > requirement and they comes under MIT license.
> >
> >
> >> 2] Currently the data that is stored in Registry is of the form wsdl.
> >> The workflow will use the data stored in the Registry .
> >> For the new workflow as there is a discussion going on what kind of data
> >> are we
> >> going to store in registry   (i.e wsdl or JSON ). This can be worked
> >> accordingly.
> >>
> >> 3] Monitoring tool : I am thinking in terms of a
> >> browser plugin or a simple java script based web based monitoring which
> >> will
> >> notify users on workflow progress in real time.
> >> This can be developed as a separate module .
> >> The Monitoring tool subscribes to a pre-specified topic to which
> >> Workflow Engine and GFac are publishing status notifications.Then the
> >> monitoring
> >> tool translates these messages and shows to to the user in the front
> end.
> >> Please see the image inserted.
> >>
> >> [image: Inline image 1]
> >>
> >> What we can also do , When user is executing a very large workflow
> >> with hundreds of variables and doesn't want to track every thing, Then
>  we
> >> can have a option to
> >> which allows customization in the monitoring tool .
> >> My point is  we can have  features in which user can monitor
> >> the variables or data he wants.
> >>
> >> Please correct me if I am wrong any where.
> >>
> >> Regards
> >> Vijayendra
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Apr 25, 2013 at 7:08 PM, Suresh Marru <smarru@apache.org>
> wrote:
> >>
> >>> On Apr 23, 2013, at 8:01 AM, Suresh Marru <smarru@apache.org> wrote:
> >>>
> >>>> Great discussion Shameera & Subho.
> >>>>
> >>>> Looks like you guys have a fair idea on what needs to be done, as you
> >>> both state, a week or two into GSOC, you can narrow down on design and
> >>> specifications and so forth. Given there are multiple students tackling
> >>> these issues collectively, if there are hard decisions to make, I would
> >>> suggest to try two approaches in parallel and quickly pick one after a
> >>> proof of concept.
> >>>
> >>> Can you all discuss the workflow composition strategies related to the
> >>> data model the same way we discussed application descriptions?
> >>>
> >>> We need to decompose the master project into smaller, well aligned
> >>> projects this weekend. So please start posting ideas on how to
> sub-divide
> >>> the projects.
> >>>
> >>> Suresh
> >>>
> >>>>
> >>>> Suresh
> >>>>
> >>>> On Apr 23, 2013, at 3:43 AM, Subho Banerjee <subs.zero@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Well exactly, as long as you can define standard way of
> communication.
> >>> That
> >>>>> is, you can define in advance what should be a string, array and
what
> >>>>> should be a integer etc. We have no problem.
> >>>>>
> >>>>> So, when you look at problems, with JSON <-> XML or the other
way
> >>> round,
> >>>>> they talk of the very general case (where you no nothing about the
> >>> data you
> >>>>> are converting other than it is valid XML/JSON). There are a myriad
> of
> >>>>> problems in that case, which you pointed out.
> >>>>>
> >>>>> But when there is standard, there is only one way of doing things,
> and
> >>> not
> >>>>> several. I think that is the way forward. So what I am proposing
is
> >>> maybe
> >>>>> we all discuss and define this standard within the first week of
GSoC
> >>>>> starting and then actually move into coding. So as long as we work
> >>> with the
> >>>>> presumption that this will be done, we really dont have to worry
a
> lot
> >>>>> about this.
> >>>>>
> >>>>> Cheers,
> >>>>> Subho.
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 23, 2013 at 11:52 AM, Shameera Rathnayaka <
> >>>>> shameerainfo@gmail.com> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On Tue, Apr 23, 2013 at 2:25 AM, Subho Banerjee <
> subs.zero@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Some of these problems are very specific to what the XML
is
> >>>>>> representing,
> >>>>>>> it might not be an actual problem in Airavata,
> >>>>>>> maybe some one more experienced with the codebase can point
this
> out.
> >>>>>>>
> >>>>>>
> >>>>>> All issues pointed out in the paper is not directly valid to
our
> >>>>>> conversion, I didn't list the issues actually need to address
in
> this
> >>> case
> >>>>>> because thought it is worth to read that introduction part which
> >>> explain
> >>>>>> the all the issues we have with this conversion and give us
a solid
> >>>>>> background of that.
> >>>>>>
> >>>>>>> 1. Anonymous values, Arrays, Implicit Typing, Character
sets -- I
> >>>>>> really
> >>>>>>> dont see these as problems, as long as you can agree that
all parts
> >>> of
> >>>>>>> airavata will treat the JSON in a standard (probably we
have to
> >>> define
> >>>>>>> this) way.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> The issue with JSON array only comes when we try to convert
XML to
> >>> JSON not
> >>>>>> the other way. If we map with JSON, inputparameters and
> >>> outputparameters in
> >>>>>> the ServiceDescription.xsd will map with JSON Arrays. Therefore
we
> >>> need to
> >>>>>> solve this issue.
> >>>>>>
> >>>>>> JSON XML JSON
> >>>>>> {"inputs":["test"]} --> <inputs>test<inputs>
 -->
> {"inputs":["test"]}
> >>>  //
> >>>>>> correct one
> >>>>>>                         --> {"inputs":"test"}     // incorrect
one
> >>>>>>
> >>>>>> 2. Namespaces, Processing Instructions -- Is this required?
> >>>>>>
> >>>>>>> Are separate namespaces used in Airavata? Only place I can
see this
> >>>>>>> being
> >>>>>>> used is probably in the WSDL, but if we can agree on another
way
> >>>>>>> of communicating registered applications' I/O parameters
to the
> >>> front
> >>>>>>> end
> >>>>>>> (JSON based), then maybe we can work around this (minor)
problem.
> >>> Are
> >>>>>>> custom processing instructions to the Xbaya XML parse even
used?
> >>>>>>> 3. Attributes -- Again, this can be fixed easily
> >>>>>>>
> >>>>>>
> >>>>>> Yes,attributes convertion will not be a big issues we can solve
it.
> As
> >>>>>> Lahiru mentioned in Hangout session namesapce handling is not
a big
> >>> issue
> >>>>>> with Airavata.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> <array name="abc">
> >>>>>>>   <element>1</element>
> >>>>>>>   <element>2</element>
> >>>>>>>   <element>3</element>
> >>>>>>>   <element>4</element>
> >>>>>>> </array>
> >>>>>>>
> >>>>>>> Can become
> >>>>>>>
> >>>>>>> {
> >>>>>>>
> >>>>>>>  abc : ['1', '2', '3', '4']
> >>>>>>>
> >>>>>>> }
> >>>>>>>
> >>>>>>
> >>>>>> With this example it show us we need to change the XML message
> format
> >>> of
> >>>>>> server side, which require to change the all schemas, If we
are
> going
> >>> to
> >>>>>> change the schemas then we need to change the way it process
it in
> >>> Ariavara
> >>>>>> core. We have dropped our initial major requirement, which is
keep
> the
> >>>>>> Airavata Server side as it is.
> >>>>>>
> >>>>>> with this conversion we only deal with json strings, yes we
can send
> >>> JSON
> >>>>>> request with other formats supported by JSON like boolen, null,
> Number
> >>>>>> etc.. But there is no way to get the same JSON from XML as XML
only
> >>> deal
> >>>>>> only with Strings. I think it is good if we can consume a this
> >>> features
> >>>>>> with JSON.
> >>>>>>
> >>>>>> let say i need to send a integer or float to the server using
JSON
> >>> then
> >>>>>> proper way is to send {"<name>":123.45} this will works
fine but
> >>> problem is
> >>>>>> how we get the same output ?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Shameera.
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Subho.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best Regards,
> >>>>>> Shameera Rathnayaka.
> >>>>>>
> >>>>>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>>>
> >>>>
> >>>
> >>>
> >>
> >
> > Thanks!
> >
> > [1] - http://people.apache.org/~smarru/papers/airavata-gce11.pdf
> > --
> > Regards
> > Andun S.L. Gunawardana
> > Undergraduate
> > Department of Computer Science And Engineering
> > University of Moratuwa
> > Sri Lanka
> >
> > Blog - http://www.insightforfuture.blogspot.com/
> > LinkedIn - http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703
> > Twitter -http://twitter.com/AndunSLG
>
>
Thanks!

-- 
Regards
Andun S.L. Gunawardana
Undergraduate
Department of Computer Science And Engineering
University of Moratuwa
Sri Lanka

Blog - http://www.insightforfuture.blogspot.com/
LinkedIn - http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703
Twitter -http://twitter.com/AndunSLG

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message