airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Subho Banerjee <subs.z...@gmail.com>
Subject Re: Airavata GSoC 2013 Master Project
Date Fri, 03 May 2013 12:43:07 GMT
Hi,
You can find a rough draft of my proposal at
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/subhobanerjee/13001

I am now working against the clock (6 hours left) to iron out the edges and
to put in any content I am missing.

Any comments would be welcome.

Cheers,
Subho,


On Fri, May 3, 2013 at 5:24 PM, Danushka Menikkumbura <
danushka.menikkumbura@gmail.com> wrote:

> Please find the proposal for "AMQP Messaging protocol support for Airavata
> WS-Messenger" at [1]
>
> Thanks,
> Danushka
>
> [1] -
>
> https://google-melange.appspot.com/gsoc/proposal/review/google/gsoc2013/danushka/1#
>
>
> On Thu, May 2, 2013 at 7:56 PM, Suresh Marru <smarru@apache.org> wrote:
>
> > Ping!! No proposals yet beyond Shameera's and Danushka's place holder.
> >
> > Suresh
> >
> > On May 1, 2013, at 5:13 PM, Suresh Marru <smarru@apache.org> wrote:
> >
> > > Hi Vijayendra,
> > >
> > > As you can see from Shameera's proposal, he proposed a JSON conversion
> > in front of WS Messenger. Also Danuska has been proposing for the AMQP
> and
> > idea and deliberating its advantages. So given all these, I would suggest
> > for you to keep focused on the UI aspects of the monitoring and write
> into
> > your proposal a plan for determining a good strategy for the plumbing to
> > WS-Eventing based existing system. You can have the concrete deliverables
> > of new UI to change colors based on executions (as it currently does in
> > XBaya), double click and show error messages and so forth. And keep it
> > exploratory for the actually messaging format.
> > >
> > > I do not have any opinion on the libraries you mentioned, but yaa a
> ajax
> > based pub system might be the right way to go. Pending the content format
> > (JSON or WS-Eventing or JMS or AMQP or something else)
> > >
> > > Suresh
> > >
> > > On May 1, 2013, at 4:13 PM, Vijayendra Grampurohit <
> > vijayendra.sdm@gmail.com> wrote:
> > >
> > >> Hi Suresh
> > >>
> > >> I am writing proposal for monitoring tool . The monitoring tool is
> > based on
> > >> pub-sub model (ws-messenger).
> > >> While writing proposal , I have to back it by technical stuff that
> tells
> > >> how can we achieve our purpose.
> > >> As this monitoring tool is supposed to be a web based , and we are
> > thinking
> > >> in the lines of
> > >> developing it in javascript.
> > >> I was looking into javascript libraries that can we used with
> > ws-messenger
> > >> in the monitoring module.
> > >> Please correct me if I am wrong.
> > >> I came across some of the libraries
> > >>
> > >>  - jQuery custom
> > >> events<
> > http://weblog.bocoup.com/publishsubscribe-with-jquery-custom-events>
> > >>  - AmplifyJS Pub/Sub <http://amplifyjs.com/api/pubsub/>
> > >>  - PubSubJS <https://github.com/mroderick/PubSubJS>
> > >>  - js-signals <http://millermedeiros.github.com/js-signals/>
> > >>
> > >> please tell me am I thinking in right direction?
> > >>
> > >> Regards
> > >> Vijayendra
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, May 1, 2013 at 5:30 PM, Suresh Marru <smarru@apache.org>
> wrote:
> > >>
> > >>> Hi Shameera,
> > >>>
> > >>> This is great, I appreciate you sharing it, I realize this is still
> > >>> working document, but I want other students to start seeing it and
> > model
> > >>> their proposals in a similar way.
> > >>>
> > >>> Airavata Mentors,
> > >>>
> > >>> Please provide feedback directly on the melange site and uncheck the
> > >>> "private" box when you comment.
> > >>>
> > >>> Suresh
> > >>>
> > >>> On May 1, 2013, at 7:52 AM, Shameera Rathnayaka <
> > shameerainfo@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi Suresh and All,
> > >>>>
> > >>>> Of course I am very much happy to share my proposal with everybody,
> > >>>> actually i was going to update this thread with the melange link
in
> > few
> > >>>> hours once i have done writing all the sections in the proposal.
I
> > >>> haven't
> > >>>> yet added the milestone plan into it and now working on it.
> > >>>>
> > >>>> The sub area i am going to work from the Master project  is '
> > >>> Implementing
> > >>>> a JSON interface to Airavata Client side and Registry component'.
> > Here is
> > >>>> the link
> > >>>>
> > >>>
> >
> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shameera/60002#
> > >>>> .
> > >>>>
> > >>>>
> > >>>> Please note that i haven't completed everything in this and still
> > doing
> > >>>> modifications .Therefore proposal content may be changed bit, need
> to
> > add
> > >>>> more technical details of the approach which explains it well.
> > >>>>
> > >>>> I would like to know the feedback from all of you regarding the
> > proposal
> > >>>> and will be modifying it if there is anything to be done. Also
> please
> > >>>> contact me if you need any help and i am very much willing to share
> my
> > >>>> thoughts with all.
> > >>>>
> > >>>> Thanks!
> > >>>> Shameera
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Wed, May 1, 2013 at 4:51 PM, Suresh Marru <smarru@apache.org>
> > wrote:
> > >>>>
> > >>>>> Hi Shameera,
> > >>>>>
> > >>>>> Excellent proposal, great job. Would you mind to make  your
> proposal
> > >>>>> public and post the link here? Your proposal should help others
to
> > look
> > >>> at
> > >>>>> it and learn from.
> > >>>>>
> > >>>>> Again I emphasize to all students, please don't feel you will
be
> > >>> competing
> > >>>>> with each others. If all of you write good proposals, there
is a
> good
> > >>>>> chance all of you will be selected. But without a good proposal,
we
> > >>> cannot
> > >>>>> help.
> > >>>>>
> > >>>>> Suresh
> > >>>>>
> > >>>>>
> > >>>>> On Apr 23, 2013, at 1:22 PM, Shameera Rathnayaka <
> > >>> shameerainfo@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> Yes it is not easy to solve all problems, But defining
our own
> > standard
> > >>>>> or
> > >>>>>> adhere to any standard
> > >>>>>> provided by third party library will solve the problem
to some
> > extend.
> > >>>>>>
> > >>>>>> Here i see two possible approaches,
> > >>>>>>
> > >>>>>> 1. Use existing third party library(we can find which is
best)
> > adhere
> > >>> to
> > >>>>> it
> > >>>>>> standard and see how we change the
> > >>>>>> backend to be inline with it.
> > >>>>>>
> > >>>>>> 2. Use our own convention with help of XMLSchema (The way
i
> > suggest).
> > >>>>>>
> > >>>>>> As Suresh mentioned we can do a POC with both approaches
to
> compare
> > >>>>>> performance
> > >>>>>> and changes need to be done in server side. Then select
the best
> > one.
> > >>>>>>
> > >>>>>> Another question was, can we works with graph data in JSON
format.
> > >>>>>> There are few JS graph framworks[1] which provide that
> > functionality.
> > >>>>>> we can use one of them to show airavata monitoring data
as graphs
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Shameera.
> > >>>>>>
> > >>>>>> [1] jqPlot <http://www.jqplot.com/index.php> , D3
<
> http://d3js.org/>
> > ,
> > >>>>>> Processing.js <http://processingjs.org> , Sencha
> > >>>>>> Charts<http://www.sencha.com/products/extjs/>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, Apr 23, 2013 at 5:44 PM, Suresh Marru <smarru@apache.org>
> > >>> wrote:
> > >>>>>>
> > >>>>>>> Hi Vijeyandra,
> > >>>>>>>
> > >>>>>>> Airavata Messaging is based on a pub-sub model and
the events
> > >>> themselves
> > >>>>>>> are xml (WS-Eventing [1]).
> > >>>>>>>
> > >>>>>>> The Messenger paper [2] should give you more information.
> > >>>>>>>
> > >>>>>>> Hi All (Especially those at WS02):
> > >>>>>>>
> > >>>>>>> Here is an old effort from a Morotuwa undergrad project,
you may
> > want
> > >>> to
> > >>>>>>> read through these papers and chat with the authors
to get
> > >>> experiences:
> > >>>>>>>
> > >>>>>>> http://dl.acm.org/citation.cfm?id=1890807
> > >>>>>>>
> > http://mgc2010.lncc.br/slides-pdf/Mooshabaya_Final_Presentation.pdf
> > >>>>>>>
> > http://kkpradeeban.blogspot.com/2010/09/mooshabaya-story-behind.html
> > >>>>>>> http://mooshabaya.wikidot.com/
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>
> >
> http://chamibuddhika.wordpress.com/2009/10/06/mooshabaya-generates-mashups-from-workflows/
> > >>>>>>>
> > >>>>>>> Suresh
> > >>>>>>> [1] - http://www.w3.org/Submission/WS-Eventing/
> > >>>>>>> [2] -
> > >>>>>>>
> > >>>>>
> > >>>
> > http://www.extreme.indiana.edu/xgws/messenger/doc/HuangY-WSMessenger.pdf
> > >>>>>>>
> > >>>>>>> On Apr 23, 2013, at 6:20 AM, Vijayendra Grampurohit
<
> > >>>>>>> vijayendra.sdm@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> Hi Suresh
> > >>>>>>>>
> > >>>>>>>> I wanted to know more about the monitoring tool
.
> > >>>>>>>> Currently from where does the monitoring tool gets
data . Is it
> > from
> > >>>>>>>> workflow interpreter ? or Is it from the WS Messenger
( that
> might
> > >>>>>>> continuously
> > >>>>>>>> send messages to monitoring tool as to tell how
much is the
> > progress
> > >>>>>>>> and what are the variables getting changed) ?
> > >>>>>>>>
> > >>>>>>>> Again the how is the data being exchanged. I guess
it must be
> xml
> > ?
> > >>>>>>>> It must be one way data exchange . I mean the data
is TO the
> > >>>>>>>> monitoring module.
> > >>>>>>>> Then monitoring Tool  is sending back this
> > >>>>>>>> data to Xbaya for displaying to the user ? Please
correct me if
> I
> > am
> > >>>>>>> wrong
> > >>>>>>>>
> > >>>>>>>> I have downloaded the source code from the trunk
. can you
> please
> > >>> point
> > >>>>>>>> me which part of code should I be code at for this
module.
> > >>>>>>>>
> > >>>>>>>> Regards
> > >>>>>>>> Vijayendra
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Tue, Apr 23, 2013 at 3:16 PM, Vijayendra Grampurohit
<
> > >>>>>>> vijayendra.sdm@gmail.com> wrote:
> > >>>>>>>> Hi
> > >>>>>>>>
> > >>>>>>>> What i am suggesting is, we send the JSON message
directly to
> > >>> Airavata
> > >>>>>>>> Backend(or Registry)
> > >>>>>>>> When the message gets build after the transport
phase, convert
> > JSON
> > >>>>>>> message
> > >>>>>>>> to SOAP(XML).
> > >>>>>>>> From that point message will treated as SOAP message.
> > >>>>>>>>
> > >>>>>>>> If we look at the JSON <--> XML conversion
there are set of
> third
> > >>> party
> > >>>>>>>> libraries we
> > >>>>>>>> can use for. But before selecting a one we need
to think about
> > >>> problems
> > >>>>>>>> having
> > >>>>>>>>
> > >>>>>>>> with JSON <--> XML and how these libraries
handle those issues.
> > >>> Because
> > >>>>>>> we
> > >>>>>>>> need a robust
> > >>>>>>>> way to do this conversions.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Shameera what you are suggesting is sending the
JSON message
> > directly
> > >>>>> to
> > >>>>>>> Registry.
> > >>>>>>>> when the message gets built after the transport
phase , convert
> > it to
> > >>>>>>> SOAP .
> > >>>>>>>>
> > >>>>>>>> If you are suggesting Registry will have JSON data
instead of
> > WSDL ,
> > >>>>>>> Then this might
> > >>>>>>>> complicate the things  for us .
> > >>>>>>>> The workflow interpreter needs wsdl(xml) to interpret
the
> > workflows
> > >>> and
> > >>>>>>> for other details .
> > >>>>>>>> Which means we might again have to do some changes
with workflow
> > >>>>>>> interpretor .
> > >>>>>>>> Yesterday from what I heard in discussion is that
, they do not
> > want
> > >>> to
> > >>>>>>> mess with workflow
> > >>>>>>>> interpreter atleast for GSOC projects.
> > >>>>>>>>
> > >>>>>>>> What I want to suggest is , why carry the  JSON
data till
> > Regisrty .
> > >>>>>>> Build a interface
> > >>>>>>>> before (Apache server API) where we  can do the
necessary
> > conversion
> > >>>>>>> (JSON  to  SOAP).
> > >>>>>>>> In this way we can avoid messing up with Airavata
server as a
> > whole.
> > >>>>>>>> Client ( using a we browser) is interacting with
JSON (web
> > service)
> > >>> but
> > >>>>>>> the Apache server
> > >>>>>>>> is interacting with SOAP.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Secondly yesterday Suresh was speaking about validating
the
> > >>> connections
> > >>>>>>> of the workflow.
> > >>>>>>>> for example , the workflow is expecting a file
as input
> > >>>>>>>> but user is giving a sting  or int .
> > >>>>>>>>
> > >>>>>>>> Here what I suggest is , while creating wsdl in
the registry
> for a
> > >>>>>>> particular
> > >>>>>>>> workflow , we can add extra information in the
form of
> > >>>>>>>> annotation as  the kind of input/ output the workflow
is
> > accepting.
> > >>>>>>>> Then we will be able to check these against users
entry during
> > >>>>> execution.
> > >>>>>>>> Please correct me if I am wrong.
> > >>>>>>>>
> > >>>>>>>> Regards
> > >>>>>>>> Vijayendra
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Tue, Apr 23, 2013 at 1:13 PM, Subho Banerjee
<
> > subs.zero@gmail.com
> > >>>>
> > >>>>>>> wrote:
> > >>>>>>>> Well exactly, as long as you can define standard
way of
> > >>> communication.
> > >>>>>>> That
> > >>>>>>>> is, you can define in advance what should be a
string, array and
> > what
> > >>>>>>>> should be a integer etc. We have no problem.
> > >>>>>>>>
> > >>>>>>>> So, when you look at problems, with JSON <->
XML or the other
> way
> > >>>>> round,
> > >>>>>>>> they talk of the very general case (where you no
nothing about
> the
> > >>> data
> > >>>>>>> you
> > >>>>>>>> are converting other than it is valid XML/JSON).
There are a
> > myriad
> > >>> of
> > >>>>>>>> problems in that case, which you pointed out.
> > >>>>>>>>
> > >>>>>>>> But when there is standard, there is only one way
of doing
> things,
> > >>> and
> > >>>>>>> not
> > >>>>>>>> several. I think that is the way forward. So what
I am proposing
> > is
> > >>>>> maybe
> > >>>>>>>> we all discuss and define this standard within
the first week of
> > GSoC
> > >>>>>>>> starting and then actually move into coding. So
as long as we
> work
> > >>> with
> > >>>>>>> the
> > >>>>>>>> presumption that this will be done, we really dont
have to
> worry a
> > >>> lot
> > >>>>>>>> about this.
> > >>>>>>>>
> > >>>>>>>> Cheers,
> > >>>>>>>> Subho.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Tue, Apr 23, 2013 at 11:52 AM, Shameera Rathnayaka
<
> > >>>>>>>> shameerainfo@gmail.com> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi,
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Apr 23, 2013 at 2:25 AM, Subho Banerjee
<
> > >>> subs.zero@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Some of these problems are very specific
to what the XML is
> > >>>>>>>>> representing,
> > >>>>>>>>>> it might not be an actual problem in Airavata,
> > >>>>>>>>>> maybe some one more experienced with the
codebase can point
> this
> > >>> out.
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> All issues pointed out in the paper is not
directly valid to
> our
> > >>>>>>>>> conversion, I didn't list the issues actually
need to address
> in
> > >>> this
> > >>>>>>> case
> > >>>>>>>>> because thought it is worth to read that introduction
part
> which
> > >>>>>>> explain
> > >>>>>>>>> the all the issues we have with this conversion
and give us a
> > solid
> > >>>>>>>>> background of that.
> > >>>>>>>>>
> > >>>>>>>>>> 1. Anonymous values, Arrays, Implicit Typing,
Character sets
> --
> > I
> > >>>>>>>>> really
> > >>>>>>>>>> dont see these as problems, as long as
you can agree that all
> > >>>>>>> parts of
> > >>>>>>>>>> airavata will treat the JSON in a standard
(probably we have
> to
> > >>>>>>> define
> > >>>>>>>>>> this) way.
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> The issue with JSON array only comes when we
try to convert XML
> > to
> > >>>>>>> JSON not
> > >>>>>>>>> the other way. If we map with JSON, inputparameters
and
> > >>>>>>> outputparameters in
> > >>>>>>>>> the ServiceDescription.xsd will map with JSON
Arrays. Therefore
> > we
> > >>>>>>> need to
> > >>>>>>>>> solve this issue.
> > >>>>>>>>>
> > >>>>>>>>> JSON XML JSON
> > >>>>>>>>> {"inputs":["test"]} --> <inputs>test<inputs>
 -->
> > >>> {"inputs":["test"]}
> > >>>>>>> //
> > >>>>>>>>> correct one
> > >>>>>>>>>                        --> {"inputs":"test"}
    // incorrect
> one
> > >>>>>>>>>
> > >>>>>>>>> 2. Namespaces, Processing Instructions -- Is
this required?
> > >>>>>>>>>
> > >>>>>>>>>> Are separate namespaces used in Airavata?
Only place I can see
> > >>>>>>> this
> > >>>>>>>>>> being
> > >>>>>>>>>> used is probably in the WSDL, but if we
can agree on another
> way
> > >>>>>>>>>> of communicating registered applications'
I/O parameters to
> the
> > >>>>>>> front
> > >>>>>>>>>> end
> > >>>>>>>>>> (JSON based), then maybe we can work around
this (minor)
> > problem.
> > >>>>>>> Are
> > >>>>>>>>>> custom processing instructions to the Xbaya
XML parse even
> used?
> > >>>>>>>>>> 3. Attributes -- Again, this can be fixed
easily
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Yes,attributes convertion will not be a big
issues we can solve
> > it.
> > >>> As
> > >>>>>>>>> Lahiru mentioned in Hangout session namesapce
handling is not a
> > big
> > >>>>>>> issue
> > >>>>>>>>> with Airavata.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> <array name="abc">
> > >>>>>>>>>>  <element>1</element>
> > >>>>>>>>>>  <element>2</element>
> > >>>>>>>>>>  <element>3</element>
> > >>>>>>>>>>  <element>4</element>
> > >>>>>>>>>> </array>
> > >>>>>>>>>>
> > >>>>>>>>>> Can become
> > >>>>>>>>>>
> > >>>>>>>>>> {
> > >>>>>>>>>>
> > >>>>>>>>>> abc : ['1', '2', '3', '4']
> > >>>>>>>>>>
> > >>>>>>>>>> }
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> With this example it show us we need to change
the XML message
> > >>> format
> > >>>>>>> of
> > >>>>>>>>> server side, which require to change the all
schemas, If we are
> > >>> going
> > >>>>>>> to
> > >>>>>>>>> change the schemas then we need to change the
way it process it
> > in
> > >>>>>>> Ariavara
> > >>>>>>>>> core. We have dropped our initial major requirement,
which is
> > keep
> > >>> the
> > >>>>>>>>> Airavata Server side as it is.
> > >>>>>>>>>
> > >>>>>>>>> with this conversion we only deal with json
strings, yes we can
> > send
> > >>>>>>> JSON
> > >>>>>>>>> request with other formats supported by JSON
like boolen, null,
> > >>> Number
> > >>>>>>>>> etc.. But there is no way to get the same JSON
from XML as XML
> > only
> > >>>>>>> deal
> > >>>>>>>>> only with Strings. I think it is good if we
can consume a this
> > >>>>> features
> > >>>>>>>>> with JSON.
> > >>>>>>>>>
> > >>>>>>>>> let say i need to send a integer or float to
the server using
> > JSON
> > >>>>> then
> > >>>>>>>>> proper way is to send {"<name>":123.45}
this will works fine
> but
> > >>>>>>> problem is
> > >>>>>>>>> how we get the same output ?
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Shameera.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Cheers,
> > >>>>>>>>>> Subho.
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> Best Regards,
> > >>>>>>>>> Shameera Rathnayaka.
> > >>>>>>>>>
> > >>>>>>>>> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Best Regards,
> > >>>>>> Shameera Rathnayaka.
> > >>>>>>
> > >>>>>> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Best Regards,
> > >>>> Shameera Rathnayaka.
> > >>>>
> > >>>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>>> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>
> > >>>
> > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message