aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erb, Stephan" <>
Subject Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.
Date Wed, 20 Apr 2016 07:26:33 GMT
So, scheduler_uri would point to a domain name with multiple A-records? 

We could probably also extend the interface to support a list of scheduler uris. That would
make an initial HA-setup simpler, as it would not require the DNS entries or a load balancer.
People could just use a list of IPs.
From: John Sirois <>
Sent: Tuesday, April 19, 2016 19:37
To: Joshua Cohen
Subject: Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.

On Mon, Apr 18, 2016 at 4:37 PM, Joshua Cohen <> wrote:

> And apparently this is not part of our fork at all, the client already
> supports this today! The only potential change that would be required would
> be ensuring the client properly follows redirects.

Aha - yes.  Thank you!

Redirect follows are indeed not enabled on the requests calls today
explicitly, but they are enabled implicitly for all non-HEAD requests IIUC.

So I think there is a small amount of work to maybe test redirection
actually works and deprecate the service discovery codepath in before
removing support.
Since there is support for the idea in the codebase already and you've
indicated no objection I'll proceed to draw up some RBs.

> On Mon, Apr 18, 2016 at 5:35 PM, Joshua Cohen <> wrote:
>> Er, it's not `proxy_url`, it's  `scheduler_uri` (which makes much more
>> sense ;)).
>> On Mon, Apr 18, 2016 at 5:34 PM, Joshua Cohen <> wrote:
>>> I'm not opposed to this, in fact we already do something similar
>>> internally. We've forked to allow configuring a `proxy_url` for
>>> each cluster. If that's present, then the client will use it rather than
>>> performing service discovery to communicate with the scheduler.
>>> On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <> wrote:
>>>> As part of work on switching the Aurora scheduler from using a copy of
>>>> the
>>>> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to
>>>> enable a
>>>> Curator feature (GUID protection [2]) that will make the scheduler more
>>>> robust to ZooKeeper outages but has the side-effect of breaking the
>>>> effectively public API formed by the ZooKeeper path names it uses to
>>>> perform leader election and service advertisement.  From Aurora's
>>>> standpoint, the API consumer is the Aurora command-line client.  It uses
>>>> service-discovery today to find the leading scheduler to communicate
>>>> with.
>>>> I propose switching the client to use HTTP re-directs instead since the
>>>> scheduler already has a redirect service and since this would reduce the
>>>> scheduler API surface area and generally move further down the road of a
>>>> pure HTTP api.
>>>> The most obvious problem I see here is just the mechanics of a proper
>>>> deprecation cycle for all those clients Aurora is not directly aware of
>>>> that rely on its service advertisment API in ZooKeeper today.
>>>> Are there other problems folks see with this?
>>>> [1]
>>>> [2]

View raw message