On Thu, Jun 12, 2014 at 11:50 PM, Supun Kamburugam= uva <supun06@gmail.com> wrote:

Hi all,

= Here is what I think about Airavata and ZooKeeper. In Airavata there are ma= ny components and these components must be stateless to achieve scalability= and reliability.Also there must be a mechanism to communicate between the = components. At the moment Airavata uses RPC calls based on Thrift for the c= ommunication.=C2=A0

ZooKeeper can be used both as a place to hold state and= as a communication layer between the components. I'm involved with a p= roject that has many distributed components like AIravata. Right now we use= Thrift services to communicate among the components. But we find it diffic= ult to use RPC calls and achieve stateless behaviour and thinking of replac= ing Thrift services with ZooKeeper based communication layer. So I think it= is better to explore the possibility of removing the Thrift services betwe= en the components and use ZooKeeper as a communication mechanism between th= e services. If you do this you will have to move the state to ZooKeeper and= will automatically achieve the stateless behaviour in the components.

Also I think trying to make ZooKeeper optional is a bad= idea. If we are trying to integrate something fundamentally important to a= rchitecture as how to store state, we shouldn't make it optional.

Thanks,
Supun..

On Thu, J= un 12, 2014 at 10:57 PM, Shameera Rathnayaka <shameerainfo@gmail.com= > wrote:

Hi Lahiru,

As i understood,=C2=A0 not only reliability , you are trying to achieve som= e other requirement by introducing zookeeper, like health monitoring of the= services, categorization with service implementation etc ... . In that cas= e, i think we can get use of zookeeper's features but if we only focus = on reliability, i have little bit of concern, why can't we use clusteri= ng + LB ?

Yes it is = better we add Zookeeper as a prerequisite if user need to use it.

<= /div>
Thanks,

Shameera.

On Thu, Jun 12, 2014 at 5:19 AM, Lahiru Gunathilake <= span dir=3D"ltr"><glahiru@gmail.com> wrote:

Hi Gagan,

I need to start another discussion about it, but I had an offline
discussion with Suresh about auto-scaling. I will start another thread
about this topic too.

Regards
Lahiru

On Wed, Jun 11, 2014 at 4:10 PM, Gagan Juneja <gagandeepjuneja@gmail.com>

wrote:

> Thanks Lahiru for pointing to nice library, added to my dictionary :).=
>
> I would like to know how are we planning to start multiple servers. > 1. Spawning new servers based on load? Some times we call it as auto > scalable.
> 2. To make some specific number of nodes available such as we want 2 > servers to be available at any time so if one goes down then I need to=
> spawn one new to make available servers count 2.
> 3. Initially start all the servers.
>
> In scenario 1 and 2 zookeeper does make sense but I don't believe = existing
> architecture support this?
>
> Regards,
> Gagan
> On 12-Jun-2014 1:19 am, "Lahiru Gunathilake" <glahiru@gmail.com> wrote= :
>
>> Hi Gagan,
>>
>> Thanks for your response. Please see my inline comments.
>>
>>
>> On Wed, Jun 11, 2014 at 3:37 PM, Gagan Juneja <gagandeepjuneja@gmail.com>
>> wrote:
>>
>>> Hi Lahiru,
>>> Just my 2 cents.
>>>
>>> I am big fan of zookeeper but also against adding multiple hop= s in the
>>> system which can add unnecessary complexity. Here I am not abl= e to
>>> understand the requirement of zookeeper may be I am wrong beca= use of less
>>> knowledge of the airavata system in whole. So I would like to = discuss
>>> following point.
>>>
>>> 1. How it will help us in making system more reliable. Zookeep= er is not
>>> able to restart services. At max it can tell whether service i= s up or not
>>> which could only be the case if airavata service goes down gra= cefully and
>>> we have any automated way to restart it. If this is just matte= r of routing
>>> client requests to the available thrift servers then this can = be achieved
>>> with the help of load balancer which I guess is already there = in thrift
>>> wish list.
>>>
>> We have multiple thrift services and currently we start only one i= nstance
>> of them and each thrift service is a stateless service. To keep th= e high
>> availability we have to start multiple instances of them in produc= tion
>> scenario. So for clients to get an available thrift service we can= use
>> zookeeper znodes to represent each available service. There are so= me
>> libraries which is doing similar[1] and I think we can use them di= rectly.
>>
>>> 2. As far as registering of different providers is concerned d= o you
>>> think for that we really need external store.
>>>
>> Yes I think so, because its light weight and reliable and we have = to do
>> very minimal amount of work to achieve all these features to Airav= ata
>> because zookeeper handle all the complexity.
>>
>>> I have seen people using zookeeper more for state management i= n
>>> distributed environments.
>>>
>> +1, we might not be the most effective users of zookeeper because = all of
>> our services are stateless services, but my point is to achieve >> fault-tolerance we can use zookeeper and with minimal work.
>>
>>> =C2=A0I would like to understand more how can we leverage zook= eeper in
>>> airavata to make system reliable.
>>>
>>>
>> [1]https://github.com/eirslett/thrift-zookeeper
>>
>>
>>
>>> Regards,
>>> Gagan
>>> On 12-Jun-2014 12:33 am, "Marlon Pierce" <marpierc@iu.edu> wrote= :
>>>
>>>> Thanks for the summary, Lahiru. I'm cc'ing the Arc= hitecture list for
>>>> additional comments.
>>>>
>>>> Marlon
>>>>
>>>> On 6/11/14 2:27 PM, Lahiru Gunathilake wrote:
>>>> > Hi All,
>>>> >
>>>> > I did little research about Apache Zookeeper[1] and h= ow to use it in
>>>> > airavata. Its really a nice way to achieve fault tole= rance and
>>>> reliable
>>>> > communication between our thrift services and clients= . Zookeeper is a
>>>> > distributed, fault tolerant system to do a reliable c= ommunication
>>>> between
>>>> > distributed applications. This is like an in-memory f= ile system which
>>>> has
>>>> > nodes in a tree structure and each node can have smal= l amount of data
>>>> > associated with it and these nodes are called znodes.= Clients can
>>>> connect
>>>> > to a zookeeper server and add/delete and update these= znodes.
>>>> >
>>>> > =C2=A0 In Apache Airavata we start multiple thrift se= rvices and these can
>>>> go
>>>> > down for maintenance or these can crash, if we use zo= okeeper to store
>>>> these
>>>> > configuration(thrift service configurations) we can a= chieve a very
>>>> reliable
>>>> > system. Basically thrift clients can dynamically disc= over available
>>>> service
>>>> > by using ephemeral znodes(Here we do not have to chan= ge the generated
>>>> > thrift client code but we have to change the location= s we are invoking
>>>> > them). ephemeral znodes will be removed when the thri= ft service goes
>>>> down
>>>> > and zookeeper guarantee the atomicity between these o= perations. With
>>>> this
>>>> > approach we can have a node hierarchy for multiple of= airavata,
>>>> > orchestrator,appcatalog and gfac thrift services.
>>>> >
>>>> > For specifically for gfac we can have different types= of services for
>>>> each
>>>> > provider implementation. This can be achieved by usin= g the
>>>> hierarchical
>>>> > support in zookeeper and providing some logic in gfac= -thrift service
>>>> to
>>>> > register it to a defined path. Using the same logic o= rchestrator can
>>>> > discover the provider specific gfac thrift service an= d route the
>>>> message to
>>>> > the correct thrift service.
>>>> >
>>>> > With this approach I think we simply have write some = client code in
>>>> thrift
>>>> > services and clients and zookeeper server installatio= n can be done as
>>>> a
>>>> > separate process and it will be easier to keep the Zo= okeeper server
>>>> > separate from Airavata because installation of Zookee= per server little
>>>> > complex in production scenario. I think we have to ma= ke sure
>>>> everything
>>>> > works fine when there is no Zookeeper running, ex: >>>> enable.zookeeper=3Dfalse
>>>> > should works fine and users doesn't have to downl= oad and start
>>>> zookeeper.
>>>> >
>>>> >
>>>> >
>>>> > [1]http://zookeeper.apache.org/
>>>> >
>>>> > Thanks
>>>> > Lahiru
>>>>
>>>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>
>

--
System Analyst Programmer
PTI Lab
Indiana University

= --
Best Regards,
Shameera Rathnayaka.=

email: shameer= a AT apache.org , shame= erainfo AT gmail.com
Blog : http://shameerarathna= yaka.blogspot.com/

--
Supun Kamburugamuva
Membe= r, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmai= l.com; =C2=A0Mobile: +1 812 369 6762
Blog: http://supun= k.blogspot.com