Mailing-List: contact dev-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
From: rick@chartbeat.com
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Subject: Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields
Message-Id: <C2DF7A39-7978-4242-925B-26A2602C451B@chartbeat.com>
Date: Sun, 12 Jun 2016 14:46:27 -0400
References: <CAJ1-todej4p9jYuz8GpnsCkaukXM40-7WGznTPeqAoSvWhfUWw@mail.gmail.com> <CAOOJoEwT4N997d2RBrqfaV4KbG-khutokw0Q3g-fgsvaDb6O7A@mail.gmail.com> <1724384244.604007.1465519462526.JavaMail.yahoo@mail.yahoo.com> <CAOOJoEyOAD=g=K4UHFb50bvBy12FQUqvRccVwH0iLU6KMCA05g@mail.gmail.com> <CAJ1-tod=xjbFKxg+ANztO0czukfUHZR=NL39J3Cy5K3MGS+bPw@mail.gmail.com>
In-Reply-To: <CAJ1-tod=xjbFKxg+ANztO0czukfUHZR=NL39J3Cy5K3MGS+bPw@mail.gmail.com>
To: dev@aurora.apache.org
archived-at: Sun, 12 Jun 2016 18:46:54 -0000

I generally shy away from technical goals that are based on a choice of lang=
uage. What is it about a go client that can't be done by extending the exist=
ing client?

> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rdelval1@binghamton.edu> wrot=
e:
>=20
> Hi David,
>=20
> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <david@dmclaughlin.com>=

> wrote:
>=20
>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>> meghdoot_b@yahoo.com.invalid> wrote:
>>=20
>>> Comments inline
>>>=20
>>>      From: David McLaughlin <dmclaughlin@apache.org>
>>> To: dev@aurora.apache.org
>>> Sent: Thursday, June 9, 2016 5:13 PM
>>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>>> mesos task related fields
>>>=20
>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rdelval1@binghamton.edu>=

>>> wrote:
>>>=20
>>>> Hello all,
>>>>=20
>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've
>> had
>>>> the pleasure of being part of the Aurora community for the last year or=

>>> so.
>>>>=20
>>>> Last year I worked to allow Aurora to utilize custom executors. With
>> the
>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
>>> became
>>>> a reality and made into Aurora late last year. (Also got to show an
>> early
>>>> beta at MesosCon!)
>>>>=20
>>>> For this year's summer, I have some new goals which I invite anyone to
>>>> provide input on.
>>>>=20
>>>> 1. Create a Golang library that works as an alternative to Aurora's
>>> Python
>>>> client and Pystachio. The initial goal of this library is to support
>> the
>>>> most critical job related Thrift API's. (Admin operations can continue
>> to
>>>> be done from the Aurora CLI). Since we support custom executors, we
>> need
>>> a
>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>> configurations to Aurora (and by extension the custom executor).
>>>=20
>>>=20
>>> You can easily add support for a different configuration format without
>>> having to rewrite the CLI in Go. You'd do that by adding your own custom=

>>> noun/verbs to the client or replacing existing commands. For example, at=

>>> Twitter we have our own version of 'aurora update start' that plugs into=

>> an
>>> internal Deploy Orchestration Service and we have a whole other command
>> for
>>> federated deploys. I can show you how we do that.
>>>=20
>>> The first thing I thought of though was - this seems like a perfect
>>> starting point to get serious about a HTTP+JSON API for Aurora. Having
>> that
>>> would make it trivial to do a CLI in any language you want, and the
>> Python
>>> CLI would really only be there to do Pystachio evaluation.
>>>>> CLI integrations is not useful for a lot of different reasons. We need=

>>> API's from other orchestration points from different components and henc=
e
>>> we would package it in a go library. Uber, I believe has taken the same
>>> route (info from this mesoscon)
>>> We are not writing a replacement cli tool. As noted admin operations
>> would
>>> heavily leverage current aurora cli.
>>> I think REST work has been postponed and attempted few times in past. We=

>>> cannot wait for that.
>>=20
>>=20
>> This is the dev list for contributions to the project, so I assumed we we=
re
>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift=

>> has Go bindings so you're all set for whatever custom tooling you're
>> building. Please use the users list for feedback on that.
>=20
>=20
> While I'm very thankful for your input on the matter, my original text ver=
y
> clearly said:
> "1. Create a Golang library that works as an *alternative* to
> Aurora's Python client and Pystachio."
>=20
> I believe every single item I have listed has the potential to become a
> useful contribution to this project.
> Therefore, I feel that it is prudent continue the discussion here (unless
> others feel similarly).
>=20
> -Renan
>=20
>>=20
>>>=20
>>>=20
>>>>=20
>>>> 2. Create support for using multiple executors within a single Aurora
>>>> scheduler instance. As of right now, only a single executor can exist
>> for
>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>> scheduler
>>>> to support multiple executors such that they can be used alongside one
>>>> another, providing flexibility for running jobs.
>>>=20
>>> Just for my own understanding - what problems are you solving with your
>>> custom executor? Sorry if you've explained this to the lists before.
>>>=20
>>> I ask because I'd really like to see Aurora stop treating the executor
>>> config as an opaque blob and being more opinionated. The
>> Thermos/Scheduler
>>> abstraction really hinders what type of user experience (UI) we can
>> build,
>>> and other frameworks do a much better job by being more opinionated and
>>> pulling executor data into the scheduler UI.
>>>>> We probably don't need to debate on this point. Executor is a first
>>> class mesos citizen and time is right for aurora to have good support fo=
r
>>> it.In our case, we have kubernetes like pods modeled through
>> docker-compose
>>> and the executor manages that. Scheduler UI main features should not get=

>>> bogged down or be held back for a particular executor. That feels
>>> incredibly restrictive. If a special UI mode for thermos executor is
>>> created, that should be fine. We do have to differentiate the scheduler
>> and
>>> thermos and aurora team has done a great job of not hard coupling the
>> two.
>>=20
>>=20
>>>>=20
>>>> 3. I'd like to add support for some first class Mesos task related
>>> fields.
>>>> These would be optional and/or have defaults so that the regular Aurora=

>>> CLI
>>>> does not break. One of the examples I'm interested in integrating is
>> the
>>>> Mesos Fetcher so that resources can be downloaded into the sandbox that=

>>> the
>>>> custom executor may need. (The executor path will never be exposed as
>>> this
>>>> will be defined serverside and be static).
>>>=20
>>>=20
>>> Wouldn't the mesos-fetcher call just be another process to be passed to
>>> your executor? I have to admit I don't know enough about how the Mesos
>>> fetcher works.
>>>>> Apache Mesos - Fetcher
>>=20
>>=20
>>=20
>>> We may add other first class fields.
>>>=20
>>>=20
>>>>=20
>>>> If anyone has any feedback on these, I'd be very glad to hear it.
>>>>=20
>>>> That's it for me for now, thanks!
>>>>=20
>>>> -Renan
>>>=20
>>>=20
>>>=20
>>>=20
>>> |
>>> |
>>> |
>>> |   |    |
>>>=20
>>>   |
>>>=20
>>>  |
>>> |
>>> |   |
>>> Apache Mesos - Fetcher
>>>   |   |
>>>=20
>>>  |
>>>=20
>>>  |
>>=20