airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijayendra Grampurohit <vijayendra....@gmail.com>
Subject Re: Airavata GSoC 2013 Master Project
Date Sun, 28 Apr 2013 10:46:26 GMT
Hi Suresh

I would like to concentrate on Workflow monitoring .

Regards
Vijayendra


On Sun, Apr 28, 2013 at 12:09 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi All,
>
> I think , defining a standard JSON message format for communication in the
> first place will
> solve the problem of projects inter dependency for a great extend. Front
> end developer
> can assume that if he generate the correct JSON format message on their
> side then rest
> will be handle correctly. By the time the person who are dealing with JSON
> thing need to
> handle it correctly.
>
> As Suresh mentioned we need to works as team while taking responsibility of
> part of
> the master project. I would like to suggest for a IRC chanel to Airavata
> then we can
> easily talk with each others through the mailing list as well as IRC
> channel too, to resolve
> problems arise in development phase.
>
> JSON <--> XML conversion is a huge part of this project because successful
> of this whole
> project depend on this. Therefore It is necessary to has a solid solution
> for this. As i explained
> in above we need to handle issues comes with this conversion like data loss
> in bidirectional
> transformation.
>
> With this model we can replace whole Registry data format to JSON, Registry
> will only deal with
> JSON data format and will send JSON Schema as application description
> instead of WSDL
> to Front end(web UI). When workflow interpreter or GFac need to talk to
> Registry, It will
> happen through JSON messages. To do that we need to convert XML message to
> JSON
> in the GFac/Workflow Interpreter pre transport phase.
>
> I would be good to get responsibility of this whole JSON thing by one
> person. Like this we
> can divide responsibility of  Workflow graph interpretation , Workflow
> composition and Workflow
> monitoring parts among us. In this case I would like to work on JSON part.
>
> Cheers,
> Shameera.
>
>
> On Sun, Apr 28, 2013 at 10:38 AM, Subho Banerjee <subs.zero@gmail.com
> >wrote:
>
> > Hi Suresh,
> >
> > I think this decomposition might be a very good way to go ahead. We can
> > apply to GSoC using these definitions and probably expand on the scope of
> > each of the projects as time progresses. However, the only problem I can
> > see is that from an implementation point of view, this might be a little
> > difficult, because none of the frontend based projects will be able test
> > themselves until the datamodel and the API is patched up to use JSON (as
> we
> > had discussed earlier).
> >
> > I would like to suggest an alternate approach (which too has certain
> > problems) where, one project deals with multiple functional units. Each
> > functional unit will try to solve a task, and will deal with the
> frontend,
> > the JSON bridge, the API as well as the data model. In this way, the
> > correctness of the code can be tested right away. Instead of waiting for
> > another project to finish something before you can move ahead. The
> problem
> > here is the enumeration of all these functional units will be a
> > very difficult and time consuming task.
> >
> > Personally I am interested in working on the (frontend) application
> > registration and workflow composition interface as well as XML <--> JSON
> > bridge which might come as a part of the workflow composition project or
> > the data-models project.
> >
> > Could you also send us a template proposal for Airavata.
> >
> > Cheers,
> > Subho.
> >
> >
> > On Sun, Apr 28, 2013 at 12:06 AM, Suresh Marru <smarru@apache.org>
> wrote:
> >
> > > Hi Guys,
> > >
> > > Does any one have any proposal on how to break the project into smaller
> > > gsoc projects?
> > >
> > > I think you all should work in parallel and as a team. There may be
> > > projects two of you can implement the same idea. Example, the workflow
> > > composer can be implemented in parallel by two students using different
> > > libraries. But we need to make sure the JSON schema and workflow model
> > > should be identical by both of them and so on.
> > >
> > > Here is one idea for the split (don't take it literally and propose
> other
> > > ideas as well):
> > >
> > > * A project fully focused on Airavata data models for application and
> > > workflows. So this person can work with rest of team to ensure a common
> > > agreed upon JSON schema is drafted and works on the conversion as
> needed
> > > with existing XML Schema.
> > > * A project focused on the Airavata API and supporting the necessary
> > > functionality needed to construct the web based interfaces. This
> project
> > > basically have to understand the current Airavata Client API and
> existing
> > > service interfaces of internal components (the REST and SOAP interfaces
> > of
> > > registry, interpreter, and messaging system -- GFac is invoked by
> > > interpreter so it is not exposed outside).
> > > * 2 projects on workflow composition interfaces themselves. These
> > projects
> > > have to interact with Airavata REST Service, parse the JSON messages,
> > > construct the graph, generate the workflow xml (XWF) at the client or a
> > > intermediary service.
> > > * A project on workflow execution interfaces. This project has to
> > > understand the current workflow inputs and workflow execution context
> and
> > > workflow representation.
> > > * A project on workflow monitoring - this project has to understand the
> > > workflow tracking schema and ws messenger services.
> > >
> > > I am not sure if this is a good way to split the projects, but I am
> just
> > > throwing an idea. Please don't look at them as separate projects. As we
> > > have been discussing right from the beginning, these interleaved tasks
> > > requires every one has to work on everything.
> > >
> > > Cheers,
> > > Suresh
> > >
> > > On Apr 27, 2013, at 1:49 PM, Andun Sameera <andunslg@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I will bring some comments which I made earlier and which will help
> in
> > > the
> > > > discussion.
> > > >
> > > > On Fri, Apr 26, 2013 at 2:09 AM, Vijayendra Grampurohit <
> > > > vijayendra.sdm@gmail.com> wrote:
> > > >
> > > >> Hi
> > > >>
> > > >> 1] Currently the workflow description has the graph representation
> > form
> > > >> xbaya .
> > > >> The data for the Workflow Inputs come from the Registry .This data
> is
> > in
> > > >> wsdl form which is stored
> > > >> in the Registry.
> > > >> There is also some metadata associated with wsdl . So that the
> client
> > > >> which has to bind the input values can use it.
> > > >>
> > > >> As we are discussing in the line of developing a  browser based
> > > >> application.
> > > >> For constructing workflow we can make use of below graph library's
> > > >> http://jsplumbtoolkit.com/jquery/demo.html
> > > >> http://raphaeljs.com/
> > > >> In our project( Openshift workflows) we had used jsplumb library
> > > >> which worked well with Angularjs .
> > > >>
> > > >> There are many more graph libraries which also can be explored.
> > > >>
> > > >
> > > > I have digged deep in to the source code via debugging to see how the
> > > XBaya
> > > > create Workflows, persists them local file system, persists them in
> > > > registry. I understood following things,
> > > >
> > > > - All the workflow design happens on top of the
> > > > org.apache.airavata.xbaya.ui.graph.GraphCanvas
> > > > - The workflow data represented via the
> > > > org.apache.airavata.workflow.model.wf.Workflow ( I think this is the
> > DAG
> > > > representation which you are talking about in Airavata Workflow
> > > > Architecture in [1])
> > > > - Local file system persistence, retrieving done via
> > > > org.apache.airavata.xbaya.core.generators.WorkflowFiler
> > > > - Registry based persistence, retrieving has done via the REST client
> > > > org.apache.airavata.rest.clientUserWorkflowResourceClient
> > > >
> > > > So If I focus on the task of implementing Web based Workflow Composer
> > for
> > > > Airavata, IMO we can reuse all three other than 1st one. 1st one have
> > to
> > > be
> > > > replace with a JavaScript based implementation. When that handles
> > design
> > > > part all the other things related to workflow can be done reusing the
> > > > others. If we use these or not, it is really valuable to understand
> the
> > > > existing way of doing the thing.
> > > >
> > > > I also looked at rapheljs and jsplumb earlier. I think they suites
> the
> > > > requirement and they comes under MIT license.
> > > >
> > > >
> > > >> 2] Currently the data that is stored in Registry is of the form
> wsdl.
> > > >> The workflow will use the data stored in the Registry .
> > > >> For the new workflow as there is a discussion going on what kind of
> > data
> > > >> are we
> > > >> going to store in registry   (i.e wsdl or JSON ). This can be worked
> > > >> accordingly.
> > > >>
> > > >> 3] Monitoring tool : I am thinking in terms of a
> > > >> browser plugin or a simple java script based web based monitoring
> > which
> > > >> will
> > > >> notify users on workflow progress in real time.
> > > >> This can be developed as a separate module .
> > > >> The Monitoring tool subscribes to a pre-specified topic to which
> > > >> Workflow Engine and GFac are publishing status notifications.Then
> the
> > > >> monitoring
> > > >> tool translates these messages and shows to to the user in the front
> > > end.
> > > >> Please see the image inserted.
> > > >>
> > > >> [image: Inline image 1]
> > > >>
> > > >> What we can also do , When user is executing a very large workflow
> > > >> with hundreds of variables and doesn't want to track every thing,
> Then
> > >  we
> > > >> can have a option to
> > > >> which allows customization in the monitoring tool .
> > > >> My point is  we can have  features in which user can monitor
> > > >> the variables or data he wants.
> > > >>
> > > >> Please correct me if I am wrong any where.
> > > >>
> > > >> Regards
> > > >> Vijayendra
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Apr 25, 2013 at 7:08 PM, Suresh Marru <smarru@apache.org>
> > > wrote:
> > > >>
> > > >>> On Apr 23, 2013, at 8:01 AM, Suresh Marru <smarru@apache.org>
> wrote:
> > > >>>
> > > >>>> Great discussion Shameera & Subho.
> > > >>>>
> > > >>>> Looks like you guys have a fair idea on what needs to be done,
as
> > you
> > > >>> both state, a week or two into GSOC, you can narrow down on design
> > and
> > > >>> specifications and so forth. Given there are multiple students
> > tackling
> > > >>> these issues collectively, if there are hard decisions to make,
I
> > would
> > > >>> suggest to try two approaches in parallel and quickly pick one
> after
> > a
> > > >>> proof of concept.
> > > >>>
> > > >>> Can you all discuss the workflow composition strategies related
to
> > the
> > > >>> data model the same way we discussed application descriptions?
> > > >>>
> > > >>> We need to decompose the master project into smaller, well aligned
> > > >>> projects this weekend. So please start posting ideas on how to
> > > sub-divide
> > > >>> the projects.
> > > >>>
> > > >>> Suresh
> > > >>>
> > > >>>>
> > > >>>> Suresh
> > > >>>>
> > > >>>> On Apr 23, 2013, at 3:43 AM, Subho Banerjee <subs.zero@gmail.com>
> > > >>> wrote:
> > > >>>>
> > > >>>>> Well exactly, as long as you can define standard way of
> > > communication.
> > > >>> That
> > > >>>>> is, you can define in advance what should be a string,
array and
> > what
> > > >>>>> should be a integer etc. We have no problem.
> > > >>>>>
> > > >>>>> So, when you look at problems, with JSON <-> XML
or the other way
> > > >>> round,
> > > >>>>> they talk of the very general case (where you no nothing
about
> the
> > > >>> data you
> > > >>>>> are converting other than it is valid XML/JSON). There
are a
> myriad
> > > of
> > > >>>>> problems in that case, which you pointed out.
> > > >>>>>
> > > >>>>> But when there is standard, there is only one way of doing
> things,
> > > and
> > > >>> not
> > > >>>>> several. I think that is the way forward. So what I am
proposing
> is
> > > >>> maybe
> > > >>>>> we all discuss and define this standard within the first
week of
> > GSoC
> > > >>>>> starting and then actually move into coding. So as long
as we
> work
> > > >>> with the
> > > >>>>> presumption that this will be done, we really dont have
to worry
> a
> > > lot
> > > >>>>> about this.
> > > >>>>>
> > > >>>>> Cheers,
> > > >>>>> Subho.
> > > >>>>>
> > > >>>>>
> > > >>>>> On Tue, Apr 23, 2013 at 11:52 AM, Shameera Rathnayaka
<
> > > >>>>> shameerainfo@gmail.com> wrote:
> > > >>>>>
> > > >>>>>> Hi,
> > > >>>>>>
> > > >>>>>> On Tue, Apr 23, 2013 at 2:25 AM, Subho Banerjee <
> > > subs.zero@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Some of these problems are very specific to what
the XML is
> > > >>>>>> representing,
> > > >>>>>>> it might not be an actual problem in Airavata,
> > > >>>>>>> maybe some one more experienced with the codebase
can point
> this
> > > out.
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> All issues pointed out in the paper is not directly
valid to our
> > > >>>>>> conversion, I didn't list the issues actually need
to address in
> > > this
> > > >>> case
> > > >>>>>> because thought it is worth to read that introduction
part which
> > > >>> explain
> > > >>>>>> the all the issues we have with this conversion and
give us a
> > solid
> > > >>>>>> background of that.
> > > >>>>>>
> > > >>>>>>> 1. Anonymous values, Arrays, Implicit Typing,
Character sets
> -- I
> > > >>>>>> really
> > > >>>>>>> dont see these as problems, as long as you can
agree that all
> > parts
> > > >>> of
> > > >>>>>>> airavata will treat the JSON in a standard (probably
we have to
> > > >>> define
> > > >>>>>>> this) way.
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> The issue with JSON array only comes when we try to
convert XML
> to
> > > >>> JSON not
> > > >>>>>> the other way. If we map with JSON, inputparameters
and
> > > >>> outputparameters in
> > > >>>>>> the ServiceDescription.xsd will map with JSON Arrays.
Therefore
> we
> > > >>> need to
> > > >>>>>> solve this issue.
> > > >>>>>>
> > > >>>>>> JSON XML JSON
> > > >>>>>> {"inputs":["test"]} --> <inputs>test<inputs>
 -->
> > > {"inputs":["test"]}
> > > >>>  //
> > > >>>>>> correct one
> > > >>>>>>                         --> {"inputs":"test"} 
   // incorrect
> one
> > > >>>>>>
> > > >>>>>> 2. Namespaces, Processing Instructions -- Is this
required?
> > > >>>>>>
> > > >>>>>>> Are separate namespaces used in Airavata? Only
place I can see
> > this
> > > >>>>>>> being
> > > >>>>>>> used is probably in the WSDL, but if we can agree
on another
> way
> > > >>>>>>> of communicating registered applications' I/O
parameters to the
> > > >>> front
> > > >>>>>>> end
> > > >>>>>>> (JSON based), then maybe we can work around this
(minor)
> problem.
> > > >>> Are
> > > >>>>>>> custom processing instructions to the Xbaya XML
parse even
> used?
> > > >>>>>>> 3. Attributes -- Again, this can be fixed easily
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> Yes,attributes convertion will not be a big issues
we can solve
> > it.
> > > As
> > > >>>>>> Lahiru mentioned in Hangout session namesapce handling
is not a
> > big
> > > >>> issue
> > > >>>>>> with Airavata.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> <array name="abc">
> > > >>>>>>>   <element>1</element>
> > > >>>>>>>   <element>2</element>
> > > >>>>>>>   <element>3</element>
> > > >>>>>>>   <element>4</element>
> > > >>>>>>> </array>
> > > >>>>>>>
> > > >>>>>>> Can become
> > > >>>>>>>
> > > >>>>>>> {
> > > >>>>>>>
> > > >>>>>>>  abc : ['1', '2', '3', '4']
> > > >>>>>>>
> > > >>>>>>> }
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> With this example it show us we need to change the
XML message
> > > format
> > > >>> of
> > > >>>>>> server side, which require to change the all schemas,
If we are
> > > going
> > > >>> to
> > > >>>>>> change the schemas then we need to change the way
it process it
> in
> > > >>> Ariavara
> > > >>>>>> core. We have dropped our initial major requirement,
which is
> keep
> > > the
> > > >>>>>> Airavata Server side as it is.
> > > >>>>>>
> > > >>>>>> with this conversion we only deal with json strings,
yes we can
> > send
> > > >>> JSON
> > > >>>>>> request with other formats supported by JSON like
boolen, null,
> > > Number
> > > >>>>>> etc.. But there is no way to get the same JSON from
XML as XML
> > only
> > > >>> deal
> > > >>>>>> only with Strings. I think it is good if we can consume
a this
> > > >>> features
> > > >>>>>> with JSON.
> > > >>>>>>
> > > >>>>>> let say i need to send a integer or float to the server
using
> JSON
> > > >>> then
> > > >>>>>> proper way is to send {"<name>":123.45} this
will works fine but
> > > >>> problem is
> > > >>>>>> how we get the same output ?
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> Shameera.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Cheers,
> > > >>>>>>> Subho.
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Best Regards,
> > > >>>>>> Shameera Rathnayaka.
> > > >>>>>>
> > > >>>>>> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > > >
> > > > Thanks!
> > > >
> > > > [1] - http://people.apache.org/~smarru/papers/airavata-gce11.pdf
> > > > --
> > > > Regards
> > > > Andun S.L. Gunawardana
> > > > Undergraduate
> > > > Department of Computer Science And Engineering
> > > > University of Moratuwa
> > > > Sri Lanka
> > > >
> > > > Blog - http://www.insightforfuture.blogspot.com/
> > > > LinkedIn -
> > http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703
> > > > Twitter -http://twitter.com/AndunSLG
> > >
> > >
> >
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message