aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Farner <wfar...@apache.org>
Subject Re: [DISCUSS] Remove dependency on ZooKeeper for leader election?
Date Tue, 18 Aug 2015 17:51:09 GMT
It's worth considering hiding the ZK dependency as a stepping stone iff
this is deemed interesting.  This would allow us focus on our strategy for
active scheduler discovery from clients, while remaining in well-trodden
territory for leader election.  We could even go so far down this path as
embedding the ZooKeeper server.

On Tuesday, August 18, 2015, Maxim Khutornenko <maxim@apache.org> wrote:

> Do you see Aurora eventually getting rid of ZK dependency entirely?
> Given that Mesos still requires ZK, I doubt we would gain much by
> rejecting the existing standard for leader election and replacing it
> with a somewhat forced storage-driven implementation.
>
> Relying on a particular DB engine implementation to enforce leader
> election protocol may be too fragile/limiting. Some engines may not
> have full support for leader election primitives. From our previous
> experience dealing with mysql locks for example, the feature may not
> work as advertised or be severely version-impaired.
>
> On Tue, Aug 18, 2015 at 10:32 AM, Joe Smith <yasumoto7@gmail.com
> <javascript:;>> wrote:
> > Awesome spelunking, since we're already dependent on the log (and you've
> gazed towards the future as well!) this sounds like a good plan.
> >
> >
> >> On Aug 18, 2015, at 10:24, Kevin Sweeney <kevints@apache.org
> <javascript:;>> wrote:
> >>
> >> While perusing the replicated log code for an upcoming talk I realized
> we
> >> could drop the use of ZooKeeper for leader election in favor of directly
> >> using the Mesos replicated log. The reason for this is that we are
> required
> >> to be the leader to write anything to the log. So the ZK-based leader
> >> election operation would be replaced with an attempt to write a noop to
> the
> >> replicated log as in [1]. This would remove one of the 3 places we use
> ZK,
> >> the others being discovery of log replicas (via libmesos, and this is
> >> already optional) and announcement to ZooKeeper for service discovery by
> >> finagle clients. This could also be made optional, as some operators use
> >> alternative load balancers.
> >>
> >> Changing the leader election logic to use the replicated log is not
> >> incongruent with other efforts to replace the hand-rolled LogStorage
> with a
> >> real SQL database - the leader election function could be implemented
> with
> >> a native database function such as [2] or [3] in any serious candidate
> >> database.
> >>
> >> [1]
> >>
> https://github.com/apache/aurora/blob/827b9abea48babe53ad5b2c521757c60f04c6dfc/src/main/java/org/apache/aurora/scheduler/log/mesos/MesosLog.java#L233-L241
> >> [2]
> >>
> http://www.postgresql.org/docs/9.1/static/functions-admin.html#FUNCTIONS-ADVISORY-LOCKS
> >> [3]
> >>
> http://dev.mysql.com/doc/refman/5.5/en/miscellaneous-functions.html#function_get-lock
>


-- 
-=Bill

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message