airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Re: Zookeeper in Airavata to achieve reliability
Date Fri, 13 Jun 2014 02:57:01 GMT
Hi Lahiru,

As i understood,  not only reliability , you are trying to achieve some
other requirement by introducing zookeeper, like health monitoring of the
services, categorization with service implementation etc ... . In that
case, i think we can get use of zookeeper's features but if we only focus
on reliability, i have little bit of concern, why can't we use clustering +
LB ?

Yes it is better we add Zookeeper as a prerequisite if user need to use it.

Thanks,
Shameera.


On Thu, Jun 12, 2014 at 5:19 AM, Lahiru Gunathilake <glahiru@gmail.com>
wrote:

> Hi Gagan,
>
> I need to start another discussion about it, but I had an offline
> discussion with Suresh about auto-scaling. I will start another thread
> about this topic too.
>
> Regards
> Lahiru
>
>
> On Wed, Jun 11, 2014 at 4:10 PM, Gagan Juneja <gagandeepjuneja@gmail.com>
> wrote:
>
> > Thanks Lahiru for pointing to nice library, added to my dictionary :).
> >
> > I would like to know how are we planning to start multiple servers.
> > 1. Spawning new servers based on load? Some times we call it as auto
> > scalable.
> > 2. To make some specific number of nodes available such as we want 2
> > servers to be available at any time so if one goes down then I need to
> > spawn one new to make available servers count 2.
> > 3. Initially start all the servers.
> >
> > In scenario 1 and 2 zookeeper does make sense but I don't believe
> existing
> > architecture support this?
> >
> > Regards,
> > Gagan
> > On 12-Jun-2014 1:19 am, "Lahiru Gunathilake" <glahiru@gmail.com> wrote:
> >
> >> Hi Gagan,
> >>
> >> Thanks for your response. Please see my inline comments.
> >>
> >>
> >> On Wed, Jun 11, 2014 at 3:37 PM, Gagan Juneja <
> gagandeepjuneja@gmail.com>
> >> wrote:
> >>
> >>> Hi Lahiru,
> >>> Just my 2 cents.
> >>>
> >>> I am big fan of zookeeper but also against adding multiple hops in the
> >>> system which can add unnecessary complexity. Here I am not able to
> >>> understand the requirement of zookeeper may be I am wrong because of
> less
> >>> knowledge of the airavata system in whole. So I would like to discuss
> >>> following point.
> >>>
> >>> 1. How it will help us in making system more reliable. Zookeeper is not
> >>> able to restart services. At max it can tell whether service is up or
> not
> >>> which could only be the case if airavata service goes down gracefully
> and
> >>> we have any automated way to restart it. If this is just matter of
> routing
> >>> client requests to the available thrift servers then this can be
> achieved
> >>> with the help of load balancer which I guess is already there in thrift
> >>> wish list.
> >>>
> >> We have multiple thrift services and currently we start only one
> instance
> >> of them and each thrift service is a stateless service. To keep the high
> >> availability we have to start multiple instances of them in production
> >> scenario. So for clients to get an available thrift service we can use
> >> zookeeper znodes to represent each available service. There are some
> >> libraries which is doing similar[1] and I think we can use them
> directly.
> >>
> >>> 2. As far as registering of different providers is concerned do you
> >>> think for that we really need external store.
> >>>
> >> Yes I think so, because its light weight and reliable and we have to do
> >> very minimal amount of work to achieve all these features to Airavata
> >> because zookeeper handle all the complexity.
> >>
> >>> I have seen people using zookeeper more for state management in
> >>> distributed environments.
> >>>
> >> +1, we might not be the most effective users of zookeeper because all of
> >> our services are stateless services, but my point is to achieve
> >> fault-tolerance we can use zookeeper and with minimal work.
> >>
> >>>  I would like to understand more how can we leverage zookeeper in
> >>> airavata to make system reliable.
> >>>
> >>>
> >> [1]https://github.com/eirslett/thrift-zookeeper
> >>
> >>
> >>
> >>> Regards,
> >>> Gagan
> >>> On 12-Jun-2014 12:33 am, "Marlon Pierce" <marpierc@iu.edu> wrote:
> >>>
> >>>> Thanks for the summary, Lahiru. I'm cc'ing the Architecture list for
> >>>> additional comments.
> >>>>
> >>>> Marlon
> >>>>
> >>>> On 6/11/14 2:27 PM, Lahiru Gunathilake wrote:
> >>>> > Hi All,
> >>>> >
> >>>> > I did little research about Apache Zookeeper[1] and how to use
it in
> >>>> > airavata. Its really a nice way to achieve fault tolerance and
> >>>> reliable
> >>>> > communication between our thrift services and clients. Zookeeper
is
> a
> >>>> > distributed, fault tolerant system to do a reliable communication
> >>>> between
> >>>> > distributed applications. This is like an in-memory file system
> which
> >>>> has
> >>>> > nodes in a tree structure and each node can have small amount of
> data
> >>>> > associated with it and these nodes are called znodes. Clients can
> >>>> connect
> >>>> > to a zookeeper server and add/delete and update these znodes.
> >>>> >
> >>>> >   In Apache Airavata we start multiple thrift services and these
can
> >>>> go
> >>>> > down for maintenance or these can crash, if we use zookeeper to
> store
> >>>> these
> >>>> > configuration(thrift service configurations) we can achieve a very
> >>>> reliable
> >>>> > system. Basically thrift clients can dynamically discover available
> >>>> service
> >>>> > by using ephemeral znodes(Here we do not have to change the
> generated
> >>>> > thrift client code but we have to change the locations we are
> invoking
> >>>> > them). ephemeral znodes will be removed when the thrift service
goes
> >>>> down
> >>>> > and zookeeper guarantee the atomicity between these operations.
With
> >>>> this
> >>>> > approach we can have a node hierarchy for multiple of airavata,
> >>>> > orchestrator,appcatalog and gfac thrift services.
> >>>> >
> >>>> > For specifically for gfac we can have different types of services
> for
> >>>> each
> >>>> > provider implementation. This can be achieved by using the
> >>>> hierarchical
> >>>> > support in zookeeper and providing some logic in gfac-thrift service
> >>>> to
> >>>> > register it to a defined path. Using the same logic orchestrator
can
> >>>> > discover the provider specific gfac thrift service and route the
> >>>> message to
> >>>> > the correct thrift service.
> >>>> >
> >>>> > With this approach I think we simply have write some client code
in
> >>>> thrift
> >>>> > services and clients and zookeeper server installation can be done
> as
> >>>> a
> >>>> > separate process and it will be easier to keep the Zookeeper server
> >>>> > separate from Airavata because installation of Zookeeper server
> little
> >>>> > complex in production scenario. I think we have to make sure
> >>>> everything
> >>>> > works fine when there is no Zookeeper running, ex:
> >>>> enable.zookeeper=false
> >>>> > should works fine and users doesn't have to download and start
> >>>> zookeeper.
> >>>> >
> >>>> >
> >>>> >
> >>>> > [1]http://zookeeper.apache.org/
> >>>> >
> >>>> > Thanks
> >>>> > Lahiru
> >>>>
> >>>>
> >>
> >>
> >> --
> >> System Analyst Programmer
> >> PTI Lab
> >> Indiana University
> >>
> >
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>



-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Mime
View raw message