airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gagan Juneja <>
Subject Re: Zookeeper in Airavata to achieve reliability
Date Wed, 11 Jun 2014 20:10:24 GMT
Thanks Lahiru for pointing to nice library, added to my dictionary :).

I would like to know how are we planning to start multiple servers.
1. Spawning new servers based on load? Some times we call it as auto
2. To make some specific number of nodes available such as we want 2
servers to be available at any time so if one goes down then I need to
spawn one new to make available servers count 2.
3. Initially start all the servers.

In scenario 1 and 2 zookeeper does make sense but I don't believe existing
architecture support this?

On 12-Jun-2014 1:19 am, "Lahiru Gunathilake" <> wrote:

> Hi Gagan,
> Thanks for your response. Please see my inline comments.
> On Wed, Jun 11, 2014 at 3:37 PM, Gagan Juneja <>
> wrote:
>> Hi Lahiru,
>> Just my 2 cents.
>> I am big fan of zookeeper but also against adding multiple hops in the
>> system which can add unnecessary complexity. Here I am not able to
>> understand the requirement of zookeeper may be I am wrong because of less
>> knowledge of the airavata system in whole. So I would like to discuss
>> following point.
>> 1. How it will help us in making system more reliable. Zookeeper is not
>> able to restart services. At max it can tell whether service is up or not
>> which could only be the case if airavata service goes down gracefully and
>> we have any automated way to restart it. If this is just matter of routing
>> client requests to the available thrift servers then this can be achieved
>> with the help of load balancer which I guess is already there in thrift
>> wish list.
> We have multiple thrift services and currently we start only one instance
> of them and each thrift service is a stateless service. To keep the high
> availability we have to start multiple instances of them in production
> scenario. So for clients to get an available thrift service we can use
> zookeeper znodes to represent each available service. There are some
> libraries which is doing similar[1] and I think we can use them directly.
>> 2. As far as registering of different providers is concerned do you think
>> for that we really need external store.
> Yes I think so, because its light weight and reliable and we have to do
> very minimal amount of work to achieve all these features to Airavata
> because zookeeper handle all the complexity.
>> I have seen people using zookeeper more for state management in
>> distributed environments.
> +1, we might not be the most effective users of zookeeper because all of
> our services are stateless services, but my point is to achieve
> fault-tolerance we can use zookeeper and with minimal work.
>>  I would like to understand more how can we leverage zookeeper in
>> airavata to make system reliable.
> [1]
>> Regards,
>> Gagan
>> On 12-Jun-2014 12:33 am, "Marlon Pierce" <> wrote:
>>> Thanks for the summary, Lahiru. I'm cc'ing the Architecture list for
>>> additional comments.
>>> Marlon
>>> On 6/11/14 2:27 PM, Lahiru Gunathilake wrote:
>>> > Hi All,
>>> >
>>> > I did little research about Apache Zookeeper[1] and how to use it in
>>> > airavata. Its really a nice way to achieve fault tolerance and reliable
>>> > communication between our thrift services and clients. Zookeeper is a
>>> > distributed, fault tolerant system to do a reliable communication
>>> between
>>> > distributed applications. This is like an in-memory file system which
>>> has
>>> > nodes in a tree structure and each node can have small amount of data
>>> > associated with it and these nodes are called znodes. Clients can
>>> connect
>>> > to a zookeeper server and add/delete and update these znodes.
>>> >
>>> >   In Apache Airavata we start multiple thrift services and these can go
>>> > down for maintenance or these can crash, if we use zookeeper to store
>>> these
>>> > configuration(thrift service configurations) we can achieve a very
>>> reliable
>>> > system. Basically thrift clients can dynamically discover available
>>> service
>>> > by using ephemeral znodes(Here we do not have to change the generated
>>> > thrift client code but we have to change the locations we are invoking
>>> > them). ephemeral znodes will be removed when the thrift service goes
>>> down
>>> > and zookeeper guarantee the atomicity between these operations. With
>>> this
>>> > approach we can have a node hierarchy for multiple of airavata,
>>> > orchestrator,appcatalog and gfac thrift services.
>>> >
>>> > For specifically for gfac we can have different types of services for
>>> each
>>> > provider implementation. This can be achieved by using the hierarchical
>>> > support in zookeeper and providing some logic in gfac-thrift service to
>>> > register it to a defined path. Using the same logic orchestrator can
>>> > discover the provider specific gfac thrift service and route the
>>> message to
>>> > the correct thrift service.
>>> >
>>> > With this approach I think we simply have write some client code in
>>> thrift
>>> > services and clients and zookeeper server installation can be done as a
>>> > separate process and it will be easier to keep the Zookeeper server
>>> > separate from Airavata because installation of Zookeeper server little
>>> > complex in production scenario. I think we have to make sure everything
>>> > works fine when there is no Zookeeper running, ex:
>>> enable.zookeeper=false
>>> > should works fine and users doesn't have to download and start
>>> zookeeper.
>>> >
>>> >
>>> >
>>> > [1]
>>> >
>>> > Thanks
>>> > Lahiru
> --
> System Analyst Programmer
> PTI Lab
> Indiana University

View raw message