airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marlon Pierce <>
Subject Re: Zookeeper in Airavata to achieve reliability
Date Wed, 11 Jun 2014 19:03:22 GMT
Thanks for the summary, Lahiru. I'm cc'ing the Architecture list for
additional comments.


On 6/11/14 2:27 PM, Lahiru Gunathilake wrote:
> Hi All,
> I did little research about Apache Zookeeper[1] and how to use it in
> airavata. Its really a nice way to achieve fault tolerance and reliable
> communication between our thrift services and clients. Zookeeper is a
> distributed, fault tolerant system to do a reliable communication between
> distributed applications. This is like an in-memory file system which has
> nodes in a tree structure and each node can have small amount of data
> associated with it and these nodes are called znodes. Clients can connect
> to a zookeeper server and add/delete and update these znodes.
>   In Apache Airavata we start multiple thrift services and these can go
> down for maintenance or these can crash, if we use zookeeper to store these
> configuration(thrift service configurations) we can achieve a very reliable
> system. Basically thrift clients can dynamically discover available service
> by using ephemeral znodes(Here we do not have to change the generated
> thrift client code but we have to change the locations we are invoking
> them). ephemeral znodes will be removed when the thrift service goes down
> and zookeeper guarantee the atomicity between these operations. With this
> approach we can have a node hierarchy for multiple of airavata,
> orchestrator,appcatalog and gfac thrift services.
> For specifically for gfac we can have different types of services for each
> provider implementation. This can be achieved by using the hierarchical
> support in zookeeper and providing some logic in gfac-thrift service to
> register it to a defined path. Using the same logic orchestrator can
> discover the provider specific gfac thrift service and route the message to
> the correct thrift service.
> With this approach I think we simply have write some client code in thrift
> services and clients and zookeeper server installation can be done as a
> separate process and it will be easier to keep the Zookeeper server
> separate from Airavata because installation of Zookeeper server little
> complex in production scenario. I think we have to make sure everything
> works fine when there is no Zookeeper running, ex: enable.zookeeper=false
> should works fine and users doesn't have to download and start zookeeper.
> [1]
> Thanks
> Lahiru

View raw message