Mailing-List: contact dev-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
MIME-Version: 1.0
Reply-To: jsirois@apache.org
In-Reply-To: <1461137192917.71566@blue-yonder.com>
References: 
 <CAC2vyCUKzxfuKtjMBnCmgF9snBdXE3eYfRHo-LE-C6D0Nie3zw@mail.gmail.com>
	<CAMnduq8qXgz=_J_50eP+6ttX7y-9TZGLGHdVsJM8yWrOEjsb4w@mail.gmail.com>
	<CAMnduq8ktkvnLAvJE-v=Rx8uK3q-7XFuPUnqB2MbPdAioNBakw@mail.gmail.com>
	<CAMnduq8nushL6mS0USPEL9+ErnTaCcS5v-UUgc6m885dZVU_Rw@mail.gmail.com>
	<CAC2vyCXW5A3F-xUDb11SNQ4n9aaZGA_tHwNVn7Gsv3hZsoW05A@mail.gmail.com>
	<1461137192917.71566@blue-yonder.com>
Date: Wed, 20 Apr 2016 01:35:59 -0600
Message-ID: 
 <CAC2vyCXsXSbfCct-EtnuJ_bsxGHocdSRz4K+cmwMCZ9XxyTksQ@mail.gmail.com>
Subject: Re: [PROPOSAL] Switch aurora client from service discovery to HTTP
 redirects.
From: John Sirois <jsirois@apache.org>
To: dev@aurora.apache.org
Cc: Joshua Cohen <jcohen@apache.org>
Content-Type: multipart/alternative; boundary=14dae9340bb91f332b0530e5a57b

--14dae9340bb91f332b0530e5a57b
Content-Type: text/plain; charset=UTF-8

On Wed, Apr 20, 2016 at 1:26 AM, Erb, Stephan <Stephan.Erb@blue-yonder.com>
wrote:

> So, scheduler_uri would point to a domain name with multiple A-records?


Yes.


>
> We could probably also extend the interface to support a list of scheduler
> uris. That would make an initial HA-setup simpler, as it would not require
> the DNS entries or a load balancer. People could just use a list of IPs.
>

I was not thinking a load balancer would be needed if the transition is
from direct zk access today, the redirect following should amount to the
same availability.  Folks that use an LB already, well they're all set.

I was waiting for someone to bring a list of IPs up.  We could do this, but
the implication is folks access the Aurora UI via a fixed ip or a name like
aurora1.mycompany.com, aurora2,mycompany.com, ...
I was having a hard time seeing how access to the UI via 1 fixed ip/name
(that could be down even though the rest are up), was fine for the UI, but
not for the client.


> ________________________________________
> From: John Sirois <jsirois@apache.org>
> Sent: Tuesday, April 19, 2016 19:37
> To: Joshua Cohen
> Cc: dev@aurora.apache.org
> Subject: Re: [PROPOSAL] Switch aurora client from service discovery to
> HTTP redirects.
>
> On Mon, Apr 18, 2016 at 4:37 PM, Joshua Cohen <jcohen@apache.org> wrote:
>
> > And apparently this is not part of our fork at all, the client already
> > supports this today! The only potential change that would be required
> would
> > be ensuring the client properly follows redirects.
> >
>
> Aha - yes.  Thank you!
>
> Redirect follows are indeed not enabled on the requests calls today
> explicitly, but they are enabled implicitly for all non-HEAD requests IIUC.
>
> So I think there is a small amount of work to maybe test redirection
> actually works and deprecate the service discovery codepath in before
> removing support.
> Since there is support for the idea in the codebase already and you've
> indicated no objection I'll proceed to draw up some RBs.
>
>
> > On Mon, Apr 18, 2016 at 5:35 PM, Joshua Cohen <jcohen@apache.org> wrote:
> >
> >> Er, it's not `proxy_url`, it's  `scheduler_uri` (which makes much more
> >> sense ;)).
> >>
> >> On Mon, Apr 18, 2016 at 5:34 PM, Joshua Cohen <jcohen@apache.org>
> wrote:
> >>
> >>> I'm not opposed to this, in fact we already do something similar
> >>> internally. We've forked clusters.py to allow configuring a
> `proxy_url` for
> >>> each cluster. If that's present, then the client will use it rather
> than
> >>> performing service discovery to communicate with the scheduler.
> >>>
> >>> On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <jsirois@apache.org>
> wrote:
> >>>
> >>>> As part of work on switching the Aurora scheduler from using a copy of
> >>>> the
> >>>> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to
> >>>> enable a
> >>>> Curator feature (GUID protection [2]) that will make the scheduler
> more
> >>>> robust to ZooKeeper outages but has the side-effect of breaking the
> >>>> effectively public API formed by the ZooKeeper path names it uses to
> >>>> perform leader election and service advertisement.  From Aurora's
> >>>> standpoint, the API consumer is the Aurora command-line client.  It
> uses
> >>>> service-discovery today to find the leading scheduler to communicate
> >>>> with.
> >>>> I propose switching the client to use HTTP re-directs instead since
> the
> >>>> scheduler already has a redirect service and since this would reduce
> the
> >>>> scheduler API surface area and generally move further down the road
> of a
> >>>> pure HTTP api.
> >>>>
> >>>> The most obvious problem I see here is just the mechanics of a proper
> >>>> deprecation cycle for all those clients Aurora is not directly aware
> of
> >>>> that rely on its service advertisment API in ZooKeeper today.
> >>>>
> >>>> Are there other problems folks see with this?
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/AURORA-1468
> >>>> [2] https://issues.apache.org/jira/browse/AURORA-1670
> >>>>
> >>>
> >>>
> >>
> >
>

--14dae9340bb91f332b0530e5a57b--