Hi Gagan,

Thanks for your response. Ple= ase see my inline comments.

On Wed, Jun 11, 2014 at 3:37 PM, Gagan Juneja <gagandeepjuneja@gmail.com> wrote:

Hi Lahiru,
Just my 2 cents.

I am big fan of zookeeper but also against adding multiple h= ops in the system which can add unnecessary complexity. Here I am not able = to understand the requirement of zookeeper may be I am wrong because of les= s knowledge of the airavata system in whole. So I would like to discuss fol= lowing point.

1. How it will help us in making system more reliable. Zooke= eper is not able to restart services. At max it can tell whether service is= up or not which could only be the case if airavata service goes down grace= fully and we have any automated way to restart it. If this is just matter o= f routing client requests to the available thrift servers then this can be = achieved with the help of load balancer which I guess is already there in t= hrift wish list.

We have multiple thrift services and currently we sta= rt only one instance of them and each thrift service is a stateless service= . To keep the high availability we have to start multiple instances of them= in production scenario. So for clients to get an available thrift service = we can use zookeeper znodes to represent each available service. There are = some libraries which is doing similar[1] and I think we can use them direct= ly.

2. As far as registering of different providers is concerned do you think f= or that we really need external store.

Yes I think so,= because its light weight and reliable and we have to do very minimal amoun= t of work to achieve all these features to Airavata because zookeeper handl= e all the complexity.

I have seen people using zookeeper more for state management= in distributed environments.

+1, we might not be the = most effective users of zookeeper because all of our services are stateless= services, but my point is to achieve fault-tolerance we can use zookeeper = and with minimal work.=C2=A0

I would like to understand more how can we leverage zookeepe= r in airavata to make system reliable.

[1]https://github.com/eirslett/thrift-zoo= keeper

=C2=A0

Regards,
Gagan

On 12-Jun-2014 12:33 am, "Marlon Pierce&quo= t; <marpierc@iu.edu= > wrote:
Thanks for the summary, Lahiru. I'm cc'ing the Architecture list fo= r
additional comments.

Marlon

On 6/11/14 2:27 PM, Lahiru Gunathilake wrote:
> Hi All,
>
> I did little research about Apache Zookeeper[1] and how to use it in > airavata. Its really a nice way to achieve fault tolerance and reliabl= e
> communication between our thrift services and clients. Zookeeper is a<= br> > distributed, fault tolerant system to do a reliable communication betw= een
> distributed applications. This is like an in-memory file system which = has
> nodes in a tree structure and each node can have small amount of data<= br> > associated with it and these nodes are called znodes. Clients can conn= ect
> to a zookeeper server and add/delete and update these znodes.
>
> =C2=A0 In Apache Airavata we start multiple thrift services and these = can go
> down for maintenance or these can crash, if we use zookeeper to store = these
> configuration(thrift service configurations) we can achieve a very rel= iable
> system. Basically thrift clients can dynamically discover available se= rvice
> by using ephemeral znodes(Here we do not have to change the generated<= br> > thrift client code but we have to change the locations we are invoking=
> them). ephemeral znodes will be removed when the thrift service goes d= own
> and zookeeper guarantee the atomicity between these operations. With t= his
> approach we can have a node hierarchy for multiple of airavata,
> orchestrator,appcatalog and gfac thrift services.
>
> For specifically for gfac we can have different types of services for = each
> provider implementation. This can be achieved by using the hierarchica= l
> support in zookeeper and providing some logic in gfac-thrift service t= o
> register it to a defined path. Using the same logic orchestrator can > discover the provider specific gfac thrift service and route the messa= ge to
> the correct thrift service.
>
> With this approach I think we simply have write some client code in th= rift
> services and clients and zookeeper server installation can be done as = a
> separate process and it will be easier to keep the Zookeeper server > separate from Airavata because installation of Zookeeper server little=
> complex in production scenario. I think we have to make sure everythin= g
> works fine when there is no Zookeeper running, ex: enable.zookeeper=3D= false
> should works fine and users doesn't have to download and start zoo= keeper.
>
>
>
> [1]http://z= ookeeper.apache.org/
>
> Thanks
> Lahiru

--
= System Analyst Programmer

PTI Lab

Indiana University