airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <sma...@apache.org>
Subject Re: Airavata GSoC 2013 Master Project
Date Tue, 30 Apr 2013 20:10:23 GMT
Hi All,

Please notify what proposal you are writing against. I would say as soon as possible create
a place holder in melange and post a link. Few more points:

* Even though your project is focusing on a topic, do not hesitate to add all the coding tasks
and milestones related to other tasks you will be contributing  to. For example Subho was
interested in workflow UI but very much interested to contribute to the JSON-XML work with
Shameera. So Subho's proposal should include all the tasks he is interested in JSON project
as well. We very well understand all these projects are inter-related so please do not shy
away from cross-linking tasks.

* Include a plan of development to follow Airavata's TDD approach - https://cwiki.apache.org/confluence/display/AIRAVATA/Tests+in+Airavata

* To make your application stronger, write briefly about how you plan to continue working
with airavata community to see your contributions can be well integrated into the main code
base and be part of the releases. Your commitment to support your contributions will be a
big plus.

* Do not forget to add details about any technical papers you plan to write based on this
GSoC work.

Good luck with your proposals, 
Suresh


On Apr 29, 2013, at 11:20 PM, Suresh Marru <smarru@apache.org> wrote:

> Hi All,
> 
> Many of you have asked about the gsoc template, I do not think we want to have any prescriptive
format and each of you can follow your style of the proposal. I just created one if its helps
- https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2013+Student+Application+Template
but don't stick to it, you can free flow yours.
> 
> But please make your proposal is public and discuss with each other. Few important points:
> 
> * You are not competing against each other. The goal is to solve a collective problem.
If one of you falls behind, it might impact the project entirely. So if you know your fellow
student will not be able to spend enough time, its equally your responsibility to make they
write those times as unavailable now.
> * Make sure all of you align the project goals. So please discuss as much as possible
within the next three days. Please do not wait for me or others on the PMC to respond. You
guys should be able to interact with each other. 
> * Directly write the proposal and edit it in melange directly. 
> * The melange proposal titles need not match the JIRA ideas page, so feel free to make
your own titles. 
> * Create public proposals and post the link here so other students can see your progress.

> * Leave the mentor blank to start with, we will assign ourselves within the next day.
> 
> Cheers,
> Suresh
> 
> On Apr 28, 2013, at 6:46 AM, Vijayendra Grampurohit <vijayendra.sdm@gmail.com>
wrote:
> 
>> Hi Suresh
>> 
>> I would like to concentrate on Workflow monitoring .
>> 
>> Regards
>> Vijayendra
>> 
>> 
>> On Sun, Apr 28, 2013 at 12:09 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>> 
>>> Hi All,
>>> 
>>> I think , defining a standard JSON message format for communication in the
>>> first place will
>>> solve the problem of projects inter dependency for a great extend. Front
>>> end developer
>>> can assume that if he generate the correct JSON format message on their
>>> side then rest
>>> will be handle correctly. By the time the person who are dealing with JSON
>>> thing need to
>>> handle it correctly.
>>> 
>>> As Suresh mentioned we need to works as team while taking responsibility of
>>> part of
>>> the master project. I would like to suggest for a IRC chanel to Airavata
>>> then we can
>>> easily talk with each others through the mailing list as well as IRC
>>> channel too, to resolve
>>> problems arise in development phase.
>>> 
>>> JSON <--> XML conversion is a huge part of this project because successful
>>> of this whole
>>> project depend on this. Therefore It is necessary to has a solid solution
>>> for this. As i explained
>>> in above we need to handle issues comes with this conversion like data loss
>>> in bidirectional
>>> transformation.
>>> 
>>> With this model we can replace whole Registry data format to JSON, Registry
>>> will only deal with
>>> JSON data format and will send JSON Schema as application description
>>> instead of WSDL
>>> to Front end(web UI). When workflow interpreter or GFac need to talk to
>>> Registry, It will
>>> happen through JSON messages. To do that we need to convert XML message to
>>> JSON
>>> in the GFac/Workflow Interpreter pre transport phase.
>>> 
>>> I would be good to get responsibility of this whole JSON thing by one
>>> person. Like this we
>>> can divide responsibility of  Workflow graph interpretation , Workflow
>>> composition and Workflow
>>> monitoring parts among us. In this case I would like to work on JSON part.
>>> 
>>> Cheers,
>>> Shameera.
>>> 
>>> 
>>> On Sun, Apr 28, 2013 at 10:38 AM, Subho Banerjee <subs.zero@gmail.com
>>>> wrote:
>>> 
>>>> Hi Suresh,
>>>> 
>>>> I think this decomposition might be a very good way to go ahead. We can
>>>> apply to GSoC using these definitions and probably expand on the scope of
>>>> each of the projects as time progresses. However, the only problem I can
>>>> see is that from an implementation point of view, this might be a little
>>>> difficult, because none of the frontend based projects will be able test
>>>> themselves until the datamodel and the API is patched up to use JSON (as
>>> we
>>>> had discussed earlier).
>>>> 
>>>> I would like to suggest an alternate approach (which too has certain
>>>> problems) where, one project deals with multiple functional units. Each
>>>> functional unit will try to solve a task, and will deal with the
>>> frontend,
>>>> the JSON bridge, the API as well as the data model. In this way, the
>>>> correctness of the code can be tested right away. Instead of waiting for
>>>> another project to finish something before you can move ahead. The
>>> problem
>>>> here is the enumeration of all these functional units will be a
>>>> very difficult and time consuming task.
>>>> 
>>>> Personally I am interested in working on the (frontend) application
>>>> registration and workflow composition interface as well as XML <-->
JSON
>>>> bridge which might come as a part of the workflow composition project or
>>>> the data-models project.
>>>> 
>>>> Could you also send us a template proposal for Airavata.
>>>> 
>>>> Cheers,
>>>> Subho.
>>>> 
>>>> 
>>>> On Sun, Apr 28, 2013 at 12:06 AM, Suresh Marru <smarru@apache.org>
>>> wrote:
>>>> 
>>>>> Hi Guys,
>>>>> 
>>>>> Does any one have any proposal on how to break the project into smaller
>>>>> gsoc projects?
>>>>> 
>>>>> I think you all should work in parallel and as a team. There may be
>>>>> projects two of you can implement the same idea. Example, the workflow
>>>>> composer can be implemented in parallel by two students using different
>>>>> libraries. But we need to make sure the JSON schema and workflow model
>>>>> should be identical by both of them and so on.
>>>>> 
>>>>> Here is one idea for the split (don't take it literally and propose
>>> other
>>>>> ideas as well):
>>>>> 
>>>>> * A project fully focused on Airavata data models for application and
>>>>> workflows. So this person can work with rest of team to ensure a common
>>>>> agreed upon JSON schema is drafted and works on the conversion as
>>> needed
>>>>> with existing XML Schema.
>>>>> * A project focused on the Airavata API and supporting the necessary
>>>>> functionality needed to construct the web based interfaces. This
>>> project
>>>>> basically have to understand the current Airavata Client API and
>>> existing
>>>>> service interfaces of internal components (the REST and SOAP interfaces
>>>> of
>>>>> registry, interpreter, and messaging system -- GFac is invoked by
>>>>> interpreter so it is not exposed outside).
>>>>> * 2 projects on workflow composition interfaces themselves. These
>>>> projects
>>>>> have to interact with Airavata REST Service, parse the JSON messages,
>>>>> construct the graph, generate the workflow xml (XWF) at the client or
a
>>>>> intermediary service.
>>>>> * A project on workflow execution interfaces. This project has to
>>>>> understand the current workflow inputs and workflow execution context
>>> and
>>>>> workflow representation.
>>>>> * A project on workflow monitoring - this project has to understand the
>>>>> workflow tracking schema and ws messenger services.
>>>>> 
>>>>> I am not sure if this is a good way to split the projects, but I am
>>> just
>>>>> throwing an idea. Please don't look at them as separate projects. As
we
>>>>> have been discussing right from the beginning, these interleaved tasks
>>>>> requires every one has to work on everything.
>>>>> 
>>>>> Cheers,
>>>>> Suresh
>>>>> 
>>>>> On Apr 27, 2013, at 1:49 PM, Andun Sameera <andunslg@gmail.com>
wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I will bring some comments which I made earlier and which will help
>>> in
>>>>> the
>>>>>> discussion.
>>>>>> 
>>>>>> On Fri, Apr 26, 2013 at 2:09 AM, Vijayendra Grampurohit <
>>>>>> vijayendra.sdm@gmail.com> wrote:
>>>>>> 
>>>>>>> Hi
>>>>>>> 
>>>>>>> 1] Currently the workflow description has the graph representation
>>>> form
>>>>>>> xbaya .
>>>>>>> The data for the Workflow Inputs come from the Registry .This
data
>>> is
>>>> in
>>>>>>> wsdl form which is stored
>>>>>>> in the Registry.
>>>>>>> There is also some metadata associated with wsdl . So that the
>>> client
>>>>>>> which has to bind the input values can use it.
>>>>>>> 
>>>>>>> As we are discussing in the line of developing a  browser based
>>>>>>> application.
>>>>>>> For constructing workflow we can make use of below graph library's
>>>>>>> http://jsplumbtoolkit.com/jquery/demo.html
>>>>>>> http://raphaeljs.com/
>>>>>>> In our project( Openshift workflows) we had used jsplumb library
>>>>>>> which worked well with Angularjs .
>>>>>>> 
>>>>>>> There are many more graph libraries which also can be explored.
>>>>>>> 
>>>>>> 
>>>>>> I have digged deep in to the source code via debugging to see how
the
>>>>> XBaya
>>>>>> create Workflows, persists them local file system, persists them
in
>>>>>> registry. I understood following things,
>>>>>> 
>>>>>> - All the workflow design happens on top of the
>>>>>> org.apache.airavata.xbaya.ui.graph.GraphCanvas
>>>>>> - The workflow data represented via the
>>>>>> org.apache.airavata.workflow.model.wf.Workflow ( I think this is
the
>>>> DAG
>>>>>> representation which you are talking about in Airavata Workflow
>>>>>> Architecture in [1])
>>>>>> - Local file system persistence, retrieving done via
>>>>>> org.apache.airavata.xbaya.core.generators.WorkflowFiler
>>>>>> - Registry based persistence, retrieving has done via the REST client
>>>>>> org.apache.airavata.rest.clientUserWorkflowResourceClient
>>>>>> 
>>>>>> So If I focus on the task of implementing Web based Workflow Composer
>>>> for
>>>>>> Airavata, IMO we can reuse all three other than 1st one. 1st one
have
>>>> to
>>>>> be
>>>>>> replace with a JavaScript based implementation. When that handles
>>>> design
>>>>>> part all the other things related to workflow can be done reusing
the
>>>>>> others. If we use these or not, it is really valuable to understand
>>> the
>>>>>> existing way of doing the thing.
>>>>>> 
>>>>>> I also looked at rapheljs and jsplumb earlier. I think they suites
>>> the
>>>>>> requirement and they comes under MIT license.
>>>>>> 
>>>>>> 
>>>>>>> 2] Currently the data that is stored in Registry is of the form
>>> wsdl.
>>>>>>> The workflow will use the data stored in the Registry .
>>>>>>> For the new workflow as there is a discussion going on what kind
of
>>>> data
>>>>>>> are we
>>>>>>> going to store in registry   (i.e wsdl or JSON ). This can be
worked
>>>>>>> accordingly.
>>>>>>> 
>>>>>>> 3] Monitoring tool : I am thinking in terms of a
>>>>>>> browser plugin or a simple java script based web based monitoring
>>>> which
>>>>>>> will
>>>>>>> notify users on workflow progress in real time.
>>>>>>> This can be developed as a separate module .
>>>>>>> The Monitoring tool subscribes to a pre-specified topic to which
>>>>>>> Workflow Engine and GFac are publishing status notifications.Then
>>> the
>>>>>>> monitoring
>>>>>>> tool translates these messages and shows to to the user in the
front
>>>>> end.
>>>>>>> Please see the image inserted.
>>>>>>> 
>>>>>>> [image: Inline image 1]
>>>>>>> 
>>>>>>> What we can also do , When user is executing a very large workflow
>>>>>>> with hundreds of variables and doesn't want to track every thing,
>>> Then
>>>>> we
>>>>>>> can have a option to
>>>>>>> which allows customization in the monitoring tool .
>>>>>>> My point is  we can have  features in which user can monitor
>>>>>>> the variables or data he wants.
>>>>>>> 
>>>>>>> Please correct me if I am wrong any where.
>>>>>>> 
>>>>>>> Regards
>>>>>>> Vijayendra
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Apr 25, 2013 at 7:08 PM, Suresh Marru <smarru@apache.org>
>>>>> wrote:
>>>>>>> 
>>>>>>>> On Apr 23, 2013, at 8:01 AM, Suresh Marru <smarru@apache.org>
>>> wrote:
>>>>>>>> 
>>>>>>>>> Great discussion Shameera & Subho.
>>>>>>>>> 
>>>>>>>>> Looks like you guys have a fair idea on what needs to
be done, as
>>>> you
>>>>>>>> both state, a week or two into GSOC, you can narrow down
on design
>>>> and
>>>>>>>> specifications and so forth. Given there are multiple students
>>>> tackling
>>>>>>>> these issues collectively, if there are hard decisions to
make, I
>>>> would
>>>>>>>> suggest to try two approaches in parallel and quickly pick
one
>>> after
>>>> a
>>>>>>>> proof of concept.
>>>>>>>> 
>>>>>>>> Can you all discuss the workflow composition strategies related
to
>>>> the
>>>>>>>> data model the same way we discussed application descriptions?
>>>>>>>> 
>>>>>>>> We need to decompose the master project into smaller, well
aligned
>>>>>>>> projects this weekend. So please start posting ideas on how
to
>>>>> sub-divide
>>>>>>>> the projects.
>>>>>>>> 
>>>>>>>> Suresh
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Suresh
>>>>>>>>> 
>>>>>>>>> On Apr 23, 2013, at 3:43 AM, Subho Banerjee <subs.zero@gmail.com>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Well exactly, as long as you can define standard
way of
>>>>> communication.
>>>>>>>> That
>>>>>>>>>> is, you can define in advance what should be a string,
array and
>>>> what
>>>>>>>>>> should be a integer etc. We have no problem.
>>>>>>>>>> 
>>>>>>>>>> So, when you look at problems, with JSON <->
XML or the other way
>>>>>>>> round,
>>>>>>>>>> they talk of the very general case (where you no
nothing about
>>> the
>>>>>>>> data you
>>>>>>>>>> are converting other than it is valid XML/JSON).
There are a
>>> myriad
>>>>> of
>>>>>>>>>> problems in that case, which you pointed out.
>>>>>>>>>> 
>>>>>>>>>> But when there is standard, there is only one way
of doing
>>> things,
>>>>> and
>>>>>>>> not
>>>>>>>>>> several. I think that is the way forward. So what
I am proposing
>>> is
>>>>>>>> maybe
>>>>>>>>>> we all discuss and define this standard within the
first week of
>>>> GSoC
>>>>>>>>>> starting and then actually move into coding. So as
long as we
>>> work
>>>>>>>> with the
>>>>>>>>>> presumption that this will be done, we really dont
have to worry
>>> a
>>>>> lot
>>>>>>>>>> about this.
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> Subho.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Tue, Apr 23, 2013 at 11:52 AM, Shameera Rathnayaka
<
>>>>>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Apr 23, 2013 at 2:25 AM, Subho Banerjee
<
>>>>> subs.zero@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Some of these problems are very specific
to what the XML is
>>>>>>>>>>> representing,
>>>>>>>>>>>> it might not be an actual problem in Airavata,
>>>>>>>>>>>> maybe some one more experienced with the
codebase can point
>>> this
>>>>> out.
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> All issues pointed out in the paper is not directly
valid to our
>>>>>>>>>>> conversion, I didn't list the issues actually
need to address in
>>>>> this
>>>>>>>> case
>>>>>>>>>>> because thought it is worth to read that introduction
part which
>>>>>>>> explain
>>>>>>>>>>> the all the issues we have with this conversion
and give us a
>>>> solid
>>>>>>>>>>> background of that.
>>>>>>>>>>> 
>>>>>>>>>>>> 1. Anonymous values, Arrays, Implicit Typing,
Character sets
>>> -- I
>>>>>>>>>>> really
>>>>>>>>>>>> dont see these as problems, as long as you
can agree that all
>>>> parts
>>>>>>>> of
>>>>>>>>>>>> airavata will treat the JSON in a standard
(probably we have to
>>>>>>>> define
>>>>>>>>>>>> this) way.
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> The issue with JSON array only comes when we
try to convert XML
>>> to
>>>>>>>> JSON not
>>>>>>>>>>> the other way. If we map with JSON, inputparameters
and
>>>>>>>> outputparameters in
>>>>>>>>>>> the ServiceDescription.xsd will map with JSON
Arrays. Therefore
>>> we
>>>>>>>> need to
>>>>>>>>>>> solve this issue.
>>>>>>>>>>> 
>>>>>>>>>>> JSON XML JSON
>>>>>>>>>>> {"inputs":["test"]} --> <inputs>test<inputs>
 -->
>>>>> {"inputs":["test"]}
>>>>>>>> //
>>>>>>>>>>> correct one
>>>>>>>>>>>                       --> {"inputs":"test"}
    // incorrect
>>> one
>>>>>>>>>>> 
>>>>>>>>>>> 2. Namespaces, Processing Instructions -- Is
this required?
>>>>>>>>>>> 
>>>>>>>>>>>> Are separate namespaces used in Airavata?
Only place I can see
>>>> this
>>>>>>>>>>>> being
>>>>>>>>>>>> used is probably in the WSDL, but if we can
agree on another
>>> way
>>>>>>>>>>>> of communicating registered applications'
I/O parameters to the
>>>>>>>> front
>>>>>>>>>>>> end
>>>>>>>>>>>> (JSON based), then maybe we can work around
this (minor)
>>> problem.
>>>>>>>> Are
>>>>>>>>>>>> custom processing instructions to the Xbaya
XML parse even
>>> used?
>>>>>>>>>>>> 3. Attributes -- Again, this can be fixed
easily
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Yes,attributes convertion will not be a big issues
we can solve
>>>> it.
>>>>> As
>>>>>>>>>>> Lahiru mentioned in Hangout session namesapce
handling is not a
>>>> big
>>>>>>>> issue
>>>>>>>>>>> with Airavata.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> <array name="abc">
>>>>>>>>>>>> <element>1</element>
>>>>>>>>>>>> <element>2</element>
>>>>>>>>>>>> <element>3</element>
>>>>>>>>>>>> <element>4</element>
>>>>>>>>>>>> </array>
>>>>>>>>>>>> 
>>>>>>>>>>>> Can become
>>>>>>>>>>>> 
>>>>>>>>>>>> {
>>>>>>>>>>>> 
>>>>>>>>>>>> abc : ['1', '2', '3', '4']
>>>>>>>>>>>> 
>>>>>>>>>>>> }
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> With this example it show us we need to change
the XML message
>>>>> format
>>>>>>>> of
>>>>>>>>>>> server side, which require to change the all
schemas, If we are
>>>>> going
>>>>>>>> to
>>>>>>>>>>> change the schemas then we need to change the
way it process it
>>> in
>>>>>>>> Ariavara
>>>>>>>>>>> core. We have dropped our initial major requirement,
which is
>>> keep
>>>>> the
>>>>>>>>>>> Airavata Server side as it is.
>>>>>>>>>>> 
>>>>>>>>>>> with this conversion we only deal with json strings,
yes we can
>>>> send
>>>>>>>> JSON
>>>>>>>>>>> request with other formats supported by JSON
like boolen, null,
>>>>> Number
>>>>>>>>>>> etc.. But there is no way to get the same JSON
from XML as XML
>>>> only
>>>>>>>> deal
>>>>>>>>>>> only with Strings. I think it is good if we can
consume a this
>>>>>>>> features
>>>>>>>>>>> with JSON.
>>>>>>>>>>> 
>>>>>>>>>>> let say i need to send a integer or float to
the server using
>>> JSON
>>>>>>>> then
>>>>>>>>>>> proper way is to send {"<name>":123.45}
this will works fine but
>>>>>>>> problem is
>>>>>>>>>>> how we get the same output ?
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Shameera.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Subho.
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Shameera Rathnayaka.
>>>>>>>>>>> 
>>>>>>>>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> [1] - http://people.apache.org/~smarru/papers/airavata-gce11.pdf
>>>>>> --
>>>>>> Regards
>>>>>> Andun S.L. Gunawardana
>>>>>> Undergraduate
>>>>>> Department of Computer Science And Engineering
>>>>>> University of Moratuwa
>>>>>> Sri Lanka
>>>>>> 
>>>>>> Blog - http://www.insightforfuture.blogspot.com/
>>>>>> LinkedIn -
>>>> http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703
>>>>>> Twitter -http://twitter.com/AndunSLG
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>> 
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>> 
> 


Mime
View raw message