airavata-architecture mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eran Chinthaka Withana <eran.chinth...@gmail.com>
Subject Re: Object Database Suggestions for Airavata Registry
Date Mon, 03 Mar 2014 00:21:12 GMT
Here is the link to hangout:
https://plus.google.com/hangouts/_/event/c1sgvk7dha37rkr0adktb195lgc?authuser=0&hl=en

Thanks,
Eran Chinthaka Withana


On Sun, Mar 2, 2014 at 12:46 PM, Suresh Marru <smarru@apache.org> wrote:

> Hi All,
>
> Since Eran has been the one who first proposed the hangout and has
> specific suggestion on this thread I prefer to postpone to 8pm (EST). But
> if others planned for 4pm, lets goahead with the plan.
>
> Any one who planned to attend now cannot make it at 8pm (EST)? If do not
> hear any objections lets shoot for 8pm. Otherwise, lets go as planned.
>
> Cheers,
> Suresh
>
> On Mar 2, 2014, at 3:31 PM, Eran Chinthaka Withana <
> eran.chinthaka@gmail.com> wrote:
>
> > Hi Suresh,
> >
> > Sorry for the late reply. I don't think I can make it at 1pm PST today.
> Can
> > we please re-schedule this to 5pm PST (8pm EST) or later?
> >
> > Thanks,
> > Eran Chinthaka Withana
> >
> >
> > On Sun, Mar 2, 2014 at 6:38 AM, Suresh Marru <smarru@apache.org> wrote:
> >
> >> Hi All,
> >>
> >> Great to see we have a good quorum. So how about 4pm EST (1pm PST) today
> >> with a hangout on air. It works best if we start a a hangout then
> (previous
> >> attempts to pre-schedules on-air events did not work well. So please
> check
> >> this mailing list around 4pm EST for the hangout on air link.
> >>
> >> Meanwhile, please join the Airavata Google Plus community, that might be
> >> easier to share the link -
> >> https://plus.google.com/communities/100700433662281905708
> >>
> >> Thanks all for willing to take time on a sunday,
> >> Suresh
> >>
> >> On Feb 28, 2014, at 9:15 PM, Supun Kamburugamuva <supun06@gmail.com>
> >> wrote:
> >>
> >>> +1 for Sunday afternoon. I can make it after 4 pm EST.
> >>>
> >>> Thanks,
> >>> Supun..
> >>>
> >>>
> >>> On Fri, Feb 28, 2014 at 5:04 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com
> >>>> wrote:
> >>>
> >>>> +1
> >>>>
> >>>> Thanks,
> >>>> Shameera.
> >>>>
> >>>>
> >>>> On Sat, Mar 1, 2014 at 3:11 AM, Eran Chinthaka Withana <
> >>>> eran.chinthaka@gmail.com> wrote:
> >>>>
> >>>>> +1 for Sunday afternoon
> >>>>>
> >>>>> Thanks,
> >>>>> Eran Chinthaka Withana
> >>>>>
> >>>>>
> >>>>> On Fri, Feb 28, 2014 at 5:17 AM, Suresh Marru <smarru@apache.org>
> >> wrote:
> >>>>>
> >>>>>> Hi Eran,
> >>>>>>
> >>>>>> This is a great idea. I myself owe few replies on this thread
and
> >>>> unable
> >>>>>> to take time to comprehend my thoughts (and realized I should
take
> >> time
> >>>>> to
> >>>>>> properly articulate the challenges otherwise we will be discussing
> >>>>>> orthogonal issues).
> >>>>>>
> >>>>>> A hangout will help us brainstorm more comprehensively. We can
have
> it
> >>>> on
> >>>>>> air so we can refer back for archival purposes. How is Sunday
> >> afternoon
> >>>>> for
> >>>>>> everyone willing to join and contribute?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Suresh
> >>>>>>
> >>>>>> On Feb 28, 2014, at 1:45 AM, Eran Chinthaka Withana <
> >>>>>> eran.chinthaka@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> Is there any chance of hosting a google hangout to talk
about
> this. I
> >>>>>> think
> >>>>>>> with long emails and multiple directions things are getting
little
> >>>> bit
> >>>>>>> confusing in thread (I'm partly responsible for this :)
). I can
> >>>> join a
> >>>>>>> video chat during a weekend but lets make sure its convenient
for
> >>>> both
> >>>>>> east
> >>>>>>> and west coasts :)
> >>>>>>>
> >>>>>>> WDYT?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Eran Chinthaka Withana
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Feb 24, 2014 at 9:32 AM, Suresh Marru <smarru@apache.org>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> I could respond to each thread in detail, but I see
the general
> >>>> sense
> >>>>> is
> >>>>>>>> inquiring on the use case, so let me try and explain
this and see
> if
> >>>>> it
> >>>>>>>> comes across. I am fully onboard with perceptions of
relational vs
> >>>>> nosql
> >>>>>>>> and also agree current Airavata needs are not a direct
map for
> NoSQL
> >>>>>>>> migration. I will summarize the driving motivation:
> >>>>>>>>
> >>>>>>>> Background: The key problem Airavata needs to solve
is getting the
> >>>> API
> >>>>>> and
> >>>>>>>> associated data model right. The problem is current
relational
> >>>>> database
> >>>>>>>> (with OpenJPA overlay) is severely limiting the API
evolution.
> >>>> Science
> >>>>>>>> Gateways by nature are very science domain and use-case
specific.
> >>>> But
> >>>>>>>> Airavata is tackling this challenging problem of providing
a
> generic
> >>>>> API
> >>>>>>>> which will meet and enable these use case centric integration.
The
> >>>>> issue
> >>>>>>>> here is, we are designing an API to handle a wide range
of known
> >>>> (and
> >>>>>> some
> >>>>>>>> foreseen) use cases. But at the same time trying to
keep it simple
> >>>> and
> >>>>>> yet
> >>>>>>>> flexible. The only way we can get through a reasonable,
normalized
> >>>>>> version
> >>>>>>>> of API is by hands-on programming against the API. Within
the
> >>>> Airavata
> >>>>>> PMC
> >>>>>>>> itself, we can solicit a half-a-dozen different ways
on how to
> >>>>> visualize
> >>>>>>>> the data model. And we need few hackethon's with real-end
users of
> >>>>>> Airavata
> >>>>>>>> until we find a common ground. All of this needs rapid
> prototyping.
> >>>>>>>> Currently a slight change in the data model is taking
close to two
> >>>>>> weeks of
> >>>>>>>> re-arcitecting the Open-JPA based registry. There are
many known
> >>>>>> problems
> >>>>>>>> with current draft of data model which have to be put-down
in the
> >>>>>> interest
> >>>>>>>> of making over all system progress.
> >>>>>>>>
> >>>>>>>> So the driving motivation is not certainly any of the
classic
> NoSQL
> >>>>>> needs.
> >>>>>>>> But a simple one, can we have registry which is schema-agnostic
> and
> >>>>> yet
> >>>>>> is
> >>>>>>>> queriable for most of the fields in the model? Can we
try 10
> >>>> different
> >>>>>>>> variants of data model (hence API) within the next 3
months with
> >>>>> focused
> >>>>>>>> hackethon's and arrive at a stable 1.0 version of API?
> >>>>>>>>
> >>>>>>>> Part one is the discussion is successful that it raised
every
> one's
> >>>>> eye
> >>>>>>>> brows. Now that we have every one's attention, what
will be a good
> >>>>> data
> >>>>>>>> store for Airavata which will meet these needs?
> >>>>>>>>
> >>>>>>>> P.S: Additional background: The API has been in development
for
> >>>> close
> >>>>>> to 3
> >>>>>>>> years and is falling short of pleasing a majority. Many
academic
> >>>>>>>> standardization efforts fail terribly trying to pretend
to
> >>>> understand
> >>>>>> all
> >>>>>>>> use cases and proposing a standard way (which ends up
> unnecessarily
> >>>>>> complex
> >>>>>>>> and not usable). Science by nature is evolutionary,
and
> restricting
> >>>>> the
> >>>>>>>> capabilities by a known set of use cases prevents the
use of
> >>>>> middleware
> >>>>>> for
> >>>>>>>> real-scientific research (and gets limited to proof
of concept
> >>>>>>>> demonstrations, papers, educational use). The only way
meeting the
> >>>>>>>> challenges of these evolving needs is to have the framework
which
> >>>> can
> >>>>>>>> evolve with minimal disruption.
> >>>>>>>>
> >>>>>>>> Great thoughts so far, please keep 'em coming until
we can find a
> >>>>>> solution
> >>>>>>>> not by the technical fancies but to address the real
need.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Suresh
> >>>>>>>>
> >>>>>>>> On Feb 24, 2014, at 11:53 AM, Lahiru Gunathilake <
> glahiru@gmail.com
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> On Mon, Feb 24, 2014 at 11:20 AM, Milinda Pathirage
<
> >>>>>>>>> milinda.pathirage@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> I also think that moving to Cassandra or any
other NoSQL will
> add
> >>>>>>>>>> unneccessary complexity to your solution. Also
designing proper
> >>>>> (easy
> >>>>>> to
> >>>>>>>>>> manage changes, easy to query) NoSQL data models
are hard
> (AFAIK,
> >>>>>>>> require
> >>>>>>>>>> lots of experience and understanding about data
structures and
> >>>>>> queries).
> >>>>>>>>>> Also migrating from one NoSQL technology to
other can require
> >>>>> complete
> >>>>>>>>>> re-write. And current relational databases can
handle heavy
> loads
> >>>>>> except
> >>>>>>>>>> Google, Twitter, Amazon and Facebook like loads.
I don't think
> >>>>>> Airavata
> >>>>>>>>>> will see Google and Amazon like loads.
> >>>>>>>>>>
> >>>>>>>>> +1
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> If the constant changes to the data model is
the problem , I
> think
> >>>>>> best
> >>>>>>>>>> option is to abstract registry implementation
to something like
> >>>>>>>> collections
> >>>>>>>>>> and resources used in WSO2 Registry [1] or something
suitable
> for
> >>>>>>>> Airavata
> >>>>>>>>>> context. That will make it easy to handle changes
in data model.
> >>>>>>>>>>
> >>>>>>>>>> Also don't let the technologies drive design
decision. Its
> always
> >>>>>>>> better to
> >>>>>>>>>> let use cases drive the design decision.
> >>>>>>>>>>
> >>>>>>>>> +1
> >>>>>>>>>
> >>>>>>>>> Regards
> >>>>>>>>> Lahiru
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Thanks
> >>>>>>>>>> Milinda
> >>>>>>>>>>
> >>>>>>>>>> [1] http://wso2.com/products/governance-registry/
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Feb 24, 2014 at 10:57 AM, Supun Kamburugamuva
<
> >>>>>>>> supun06@gmail.com
> >>>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi all,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm not trying to discourage you on your
exploration to NoSQL
> >>>>>>>> databases.
> >>>>>>>>>> I
> >>>>>>>>>>> have the following concern.
> >>>>>>>>>>>
> >>>>>>>>>>> Your database schema is moderately complex
- even for a RDBMS
> it
> >>>>>> seems
> >>>>>>>>>>> complex and the data size is relatively
small. I'm not sure
> about
> >>>>> the
> >>>>>>>>>>> current tools available but I think you
will need to write more
> >>>>> code
> >>>>>> to
> >>>>>>>>>>> support all your requirements in a NoSQL
database. So writing
> >>>> more
> >>>>>> code
> >>>>>>>>>> and
> >>>>>>>>>>> allow redundancy to support *relatively
small* and *structured
> >>>>>>>>>>> data*doesn't seem right to me. May be I'm
wrong and there are
> >>>>> better
> >>>>>>>>>>> tools in
> >>>>>>>>>>> NoSQL than RDBMS, which I doubt.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Supun..
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Feb 23, 2014 at 5:20 PM, Suresh
Marru <
> smarru@apache.org
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Airavata is actively migrating to use
Thrift API for the
> >>>> RESTless
> >>>>>>>>>> design
> >>>>>>>>>>>> and to facilitate various language bindings
from client
> >>>> gateways.
> >>>>>> The
> >>>>>>>>>>>> programming language support in thrift
has been so far very
> >>>>>>>>>> encouraging.
> >>>>>>>>>>>> The current architecture is looking
like Figure 1 at [1].
> >>>>>>>>>>>>
> >>>>>>>>>>>> Language specific clients will be released
as thrift SDK's
> >>>>> (similar
> >>>>>> to
> >>>>>>>>>>>> evernote sdk's [1]). These clients will
be integrated into
> >>>> gateway
> >>>>>>>>>>> portals
> >>>>>>>>>>>> which connect to the API Server. The
API operations brokers he
> >>>>>> simple
> >>>>>>>>>>> calls
> >>>>>>>>>>>> into one or more backend CPI calls (Airavata
internal
> component
> >>>>>>>>>>>> interfaces).  An example set of mappings
are illustrated in
> >>>>> Figure 2
> >>>>>>>> at
> >>>>>>>>>>>> [1]. The current draft of thrift API
for version 0.12 is at
> [3],
> >>>>>>>> please
> >>>>>>>>>>> pay
> >>>>>>>>>>>> attention to experiment model at [4].
> >>>>>>>>>>>>
> >>>>>>>>>>>> For the persistent store, we had few
iterations of Airavata
> >>>>> Registry
> >>>>>>>>>>>> shifting from a legacy XRegistry to
JackRabbit to now a
> OpenJPA
> >>>>>> based
> >>>>>>>>>>>> registry. To allow the API and the associated
data models to
> >>>>> evolve,
> >>>>>>>> it
> >>>>>>>>>>>> will be useful to explore object databases
so we can store the
> >>>>>>>>>> serialized
> >>>>>>>>>>>> version of thrift objects directly.
But it will be nice to
> have
> >>>>> all
> >>>>>>>> (or
> >>>>>>>>>>>> most) of the fields queriable. This
calls for a more
> >>>> column-family
> >>>>>>>>>> design
> >>>>>>>>>>>> of any NoSQL approaches.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Any recommendations for a registry architecture?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Quickly hacking through I find the following
approach a viable
> >>>>> one:
> >>>>>>>>>>>> ZombieDB[5] over astyanax[6] which talks
to Cassandra.
> Airavata
> >>>>> can
> >>>>>>>>>>> benefit
> >>>>>>>>>>>> immediately from the replication and
reliability of cassandra
> >>>> and
> >>>>>>>>>>>> scalability in near future. Some of
the model objects like
> >>>>>> experiment
> >>>>>>>>>>>> creation will need to have strong consistency
and most of the
> >>>>>>>>>> monitoring
> >>>>>>>>>>>> can live with eventual consistency.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Critical comments please?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for your time,
> >>>>>>>>>>>> Suresh
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1] -
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/AIRAVATA/2014/02/23/Brainstorming+Diagrams
> >>>>>>>>>>>> [2] - https://dev.evernote.com/doc/
> >>>>>>>>>>>> [3] -
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://git-wip-us.apache.org/repos/asf?p=airavata.git;a=tree;f=airavata-api/thrift-interface-descriptions;hb=HEAD
> >>>>>>>>>>>> [4] -
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://git-wip-us.apache.org/repos/asf?p=airavata.git;a=blob_plain;f=airavata-api/thrift-interface-descriptions/experimentModel.thrift;hb=HEAD
> >>>>>>>>>>>> [5] - https://github.com/MisterTea/ZombieDB
> >>>>>>>>>>>> [6] - https://github.com/Netflix/astyanax
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Supun Kamburugamuva
> >>>>>>>>>>> Member, Apache Software Foundation; http://www.apache.org
> >>>>>>>>>>> E-mail: supun06@gmail.com;  Mobile: +1 812
369 6762
> >>>>>>>>>>> Blog: http://supunk.blogspot.com
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Milinda Pathirage
> >>>>>>>>>> PhD Student Indiana University, Bloomington;
> >>>>>>>>>> E-mail: milinda.pathirage@gmail.com
> >>>>>>>>>> Web: http://mpathirage.com
> >>>>>>>>>> Blog: http://blog.mpathirage.com
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> System Analyst Programmer
> >>>>>>>>> PTI Lab
> >>>>>>>>> Indiana University
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best Regards,
> >>>> Shameera Rathnayaka.
> >>>>
> >>>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Supun Kamburugamuva
> >>> Member, Apache Software Foundation; http://www.apache.org
> >>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> Blog: http://supunk.blogspot.com
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message