Mailing-List: contact dev-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
Content-Type: text/plain;
	charset=us-ascii
Mime-Version: 1.0 (1.0)
Subject: Re: Aurora now supports multiple executors
From: meghdoot_b@yahoo.com.INVALID
In-Reply-To: <CANMwFmVwHHbEA1MPsKHG25-9kuPjhhzpgq-FtWgwOLe7=cROtg@mail.gmail.com>
Date: Fri, 5 Aug 2016 10:15:20 -0700
Cc: Renan Delvalle <rdelval1@binghamton.edu>
Content-Transfer-Encoding: quoted-printable
Message-Id: <3C861F60-B72B-4E02-99C4-A737FA10DAFA@yahoo.com>
References: <CAJ1-todej4p9jYuz8GpnsCkaukXM40-7WGznTPeqAoSvWhfUWw@mail.gmail.com> <CAOOJoEwT4N997d2RBrqfaV4KbG-khutokw0Q3g-fgsvaDb6O7A@mail.gmail.com> <1724384244.604007.1465519462526.JavaMail.yahoo@mail.yahoo.com> <CAOOJoEyOAD=g=K4UHFb50bvBy12FQUqvRccVwH0iLU6KMCA05g@mail.gmail.com> <CAJ1-tod=xjbFKxg+ANztO0czukfUHZR=NL39J3Cy5K3MGS+bPw@mail.gmail.com> <C2DF7A39-7978-4242-925B-26A2602C451B@chartbeat.com> <CAOTkfX7gXVeqpR6b80C0P25atumV3cQ_H6JPF=PpWJ8XViyj=Q@mail.gmail.com> <0E9A9533-B7F1-4F4E-A648-068F7E035603@yahoo.com> <CAOTkfX4d1iF7T3HHWhpMKudaS_PJhE++rEyatj9ttAyfx=dyBw@mail.gmail.com> <78B315E4-E919-49C0-99C8-EE1BDEE8BE78@yahoo.com> <CAOOJoEyDEetEqQGLRcryySnTFkvFbjsoQHNY+P5yvi0aHOjeFw@mail.gmail.com> <360755398.11496618.1470382535604.JavaMail.yahoo@mail.yahoo.com> <CANMwFmVwHHbEA1MPsKHG25-9kuPjhhzpgq-FtWgwOLe7=cROtg@mail.gmail.com>
To: "dev@aurora.apache.org" <dev@aurora.apache.org>
archived-at: Fri, 05 Aug 2016 17:15:38 -0000

Expect that to be up by mid next week and put in 0.16 docs.

Sent from my iPhone

> On Aug 5, 2016, at 9:23 AM, Zhitao Li <zhitaoli.cs@gmail.com> wrote:
>=20
> This sounds awesome!
>=20
> Is there a document for how to use the feature?
>=20
> On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya <
> meghdoot_b@yahoo.com.invalid> wrote:
>=20
>> Renaming the subject.
>> Renan's latest patch in 0.16 has completed the goal of implementing both 2=

>> and 3. So, Aurora now can not only support custom executors but run
>> multiple executors in the cluster!! Mesos fetcher has been integrated as
>> well to help the custom executors have access to downloaded artifacts tha=
t
>> it needs. We are going to add dedicated documentation on how to enable th=
is
>> in 0.16.
>> This was 2 summers in making with Renan's internship slots. Special thank=
s
>> to Maxim, Joshua and Bill to make this happen and mentoring Renan in the
>> process.
>>=20
>>=20
>> As for point 1, we are going to put out the golang client in few weeks to=

>> easily test different executors (including thermos) with Aurora with real=

>> examples. During the same time we will release production grade
>> docker-compose executor simulating running container pods with Aurora.
>> Expect a detailed blog in a month.
>> Thx
>>      From: David McLaughlin <dmclaughlin@apache.org>
>> To: dev@aurora.apache.org
>> Sent: Monday, June 13, 2016 10:39 AM
>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>> mesos task related fields
>>=20
>> Thanks, also +1 to supporting points 2 and 3.
>>=20
>>> On Mon, Jun 13, 2016 at 10:33 AM, <meghdoot_b@yahoo.com.invalid> wrote:
>>>=20
>>> Got it. Thx Max.
>>> Renan will starting working on 2 and 3, involving and updating you all.
>>>=20
>>> Thx
>>>=20
>>> Sent from my iPhone
>>>=20
>>>>> On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <maxim@apache.org>
>>>> wrote:
>>>>=20
>>>> David pinged me offline suggesting I misread the proposal for (2) as
>>>> an attempt to support multiple executors per task rather than "per
>>>> cluster". If that's the case, I am happy to change my vote for (2) to
>>>> +1. I don't see much harm (aside from possible UI inconsistencies)
>>>> from multiple executor types sharing the same cluster.
>>>>=20
>>>>> On Mon, Jun 13, 2016 at 9:54 AM,  <meghdoot_b@yahoo.com.invalid>
>> wrote:
>>>>> I will leave Renan for a detailed response.
>>>>>=20
>>>>> If aurora can schedule Job A with thermos and Job B that requires a
>>> different executor that is a first class concept in mesos, why is there a=

>>> objection? Renan's patch of replacing thermos executor is already in
>>> aurora. However right now aurora can statically configure only one
>>> executor. And so we want a job submission dynamically can call out the
>>> executor type.
>>>>> Please clarify your views.
>>>>> Right now the docker integration story in mesos is weak (current and
>>> also the new universal containerizer). We see much value in using
>> executors
>>> like
>>>>> https://github.com/mesos/docker-compose-executor
>>>>> With aurora jobs.
>>>>>=20
>>>>> Points 2 and points 3 make the custom executor story stronger. (For
>>> example we will run both thermos and compose executor depending on job
>> type)
>>>>>=20
>>>>>=20
>>>>> Point 1, I agree. That is outside of aurora and most likely will be a
>>> shim on top of generated go thrift bindings but a helper to easily test
>>> custom executors.
>>>>> Last time when we reviewed with aurora team, it came out current
>> aurora
>>> cli is tightly coupled with thermos objects for jobs and hence keep it a=
s
>>> is.
>>>>>=20
>>>>> Thx
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> Sent from my iPhone
>>>>>=20
>>>>>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <maxim@apache.org>
>>> wrote:
>>>>>>=20
>>>>>> I am with Rick and David on this. The (1) and (2) feel like building
>> a
>>>>>> parallel universe within Aurora without a clear justification on the
>>>>>> long-term benefits for the community.
>>>>>>=20
>>>>>> I am strong -1 on bringing yet another language into the Aurora
>>>>>> ecosystem without identifying the strategic upside. As David
>>>>>> suggested, any efforts towards building the first-class REST API
>>>>>> instead would go a long way and will be much appreciated by everyone.=

>>>>>>=20
>>>>>> I also don't see a reason to support multiple executors per task.
>> That
>>>>>> feels like re-creating thermos in a thermos-less world.
>>>>>>=20
>>>>>> As for (3), I'd like to see more details on the mechanics here but
>>>>>> generally positive towards supporting more TaskInfo features in
>>>>>> Aurora.
>>>>>>=20
>>>>>>=20
>>>>>>> On Sun, Jun 12, 2016 at 11:46 AM,  <rick@chartbeat.com> wrote:
>>>>>>> I generally shy away from technical goals that are based on a choice=

>>> of language. What is it about a go client that can't be done by extendin=
g
>>> the existing client?
>>>>>>>=20
>>>>>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <
>> rdelval1@binghamton.edu>
>>> wrote:
>>>>>>>>=20
>>>>>>>> Hi David,
>>>>>>>>=20
>>>>>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <
>>> david@dmclaughlin.com>
>>>>>>>> wrote:
>>>>>>>>=20
>>>>>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>>>>>>>>> meghdoot_b@yahoo.com.invalid> wrote:
>>>>>>>>>=20
>>>>>>>>>> Comments inline
>>>>>>>>>>=20
>>>>>>>>>>   From: David McLaughlin <dmclaughlin@apache.org>
>>>>>>>>>> To: dev@aurora.apache.org
>>>>>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
>>>>>>>>>> Subject: Re: Golang Aurora lib, multiple executor support,
>>> integrate
>>>>>>>>>> mesos task related fields
>>>>>>>>>>=20
>>>>>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <
>>> rdelval1@binghamton.edu>
>>>>>>>>>> wrote:
>>>>>>>>>>=20
>>>>>>>>>>> Hello all,
>>>>>>>>>>>=20
>>>>>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and
>>> I've
>>>>>>>>> had
>>>>>>>>>>> the pleasure of being part of the Aurora community for the last
>>> year or
>>>>>>>>>> so.
>>>>>>>>>>>=20
>>>>>>>>>>> Last year I worked to allow Aurora to utilize custom executors.
>>> With
>>>>>>>>> the
>>>>>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that
>>> feature
>>>>>>>>>> became
>>>>>>>>>>> a reality and made into Aurora late last year. (Also got to show=

>>> an
>>>>>>>>> early
>>>>>>>>>>> beta at MesosCon!)
>>>>>>>>>>>=20
>>>>>>>>>>> For this year's summer, I have some new goals which I invite
>>> anyone to
>>>>>>>>>>> provide input on.
>>>>>>>>>>>=20
>>>>>>>>>>> 1. Create a Golang library that works as an alternative to
>>> Aurora's
>>>>>>>>>> Python
>>>>>>>>>>> client and Pystachio. The initial goal of this library is to
>>> support
>>>>>>>>> the
>>>>>>>>>>> most critical job related Thrift API's. (Admin operations can
>>> continue
>>>>>>>>> to
>>>>>>>>>>> be done from the Aurora CLI). Since we support custom executors,=

>>> we
>>>>>>>>> need
>>>>>>>>>> a
>>>>>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs=

>>>>>>>>>>> configurations to Aurora (and by extension the custom executor).=

>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> You can easily add support for a different configuration format
>>> without
>>>>>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own=

>>> custom
>>>>>>>>>> noun/verbs to the client or replacing existing commands. For
>>> example, at
>>>>>>>>>> Twitter we have our own version of 'aurora update start' that
>>> plugs into
>>>>>>>>> an
>>>>>>>>>> internal Deploy Orchestration Service and we have a whole other
>>> command
>>>>>>>>> for
>>>>>>>>>> federated deploys. I can show you how we do that.
>>>>>>>>>>=20
>>>>>>>>>> The first thing I thought of though was - this seems like a
>> perfect
>>>>>>>>>> starting point to get serious about a HTTP+JSON API for Aurora.
>>> Having
>>>>>>>>> that
>>>>>>>>>> would make it trivial to do a CLI in any language you want, and
>> the
>>>>>>>>> Python
>>>>>>>>>> CLI would really only be there to do Pystachio evaluation.
>>>>>>>>>>>> CLI integrations is not useful for a lot of different reasons.
>>> We need
>>>>>>>>>> API's from other orchestration points from different components
>>> and hence
>>>>>>>>>> we would package it in a go library. Uber, I believe has taken
>> the
>>> same
>>>>>>>>>> route (info from this mesoscon)
>>>>>>>>>> We are not writing a replacement cli tool. As noted admin
>>> operations
>>>>>>>>> would
>>>>>>>>>> heavily leverage current aurora cli.
>>>>>>>>>> I think REST work has been postponed and attempted few times in
>>> past. We
>>>>>>>>>> cannot wait for that.
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> This is the dev list for contributions to the project, so I
>> assumed
>>> we were
>>>>>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and=

>>> Thrift
>>>>>>>>> has Go bindings so you're all set for whatever custom tooling
>> you're
>>>>>>>>> building. Please use the users list for feedback on that.
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> While I'm very thankful for your input on the matter, my original
>>> text very
>>>>>>>> clearly said:
>>>>>>>> "1. Create a Golang library that works as an *alternative* to
>>>>>>>> Aurora's Python client and Pystachio."
>>>>>>>>=20
>>>>>>>> I believe every single item I have listed has the potential to
>>> become a
>>>>>>>> useful contribution to this project.
>>>>>>>> Therefore, I feel that it is prudent continue the discussion here
>>> (unless
>>>>>>>> others feel similarly).
>>>>>>>>=20
>>>>>>>> -Renan
>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> 2. Create support for using multiple executors within a single
>>> Aurora
>>>>>>>>>>> scheduler instance. As of right now, only a single executor can
>>> exist
>>>>>>>>> for
>>>>>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>>>>>>>>> scheduler
>>>>>>>>>>> to support multiple executors such that they can be used
>>> alongside one
>>>>>>>>>>> another, providing flexibility for running jobs.
>>>>>>>>>>=20
>>>>>>>>>> Just for my own understanding - what problems are you solving
>> with
>>> your
>>>>>>>>>> custom executor? Sorry if you've explained this to the lists
>>> before.
>>>>>>>>>>=20
>>>>>>>>>> I ask because I'd really like to see Aurora stop treating the
>>> executor
>>>>>>>>>> config as an opaque blob and being more opinionated. The
>>>>>>>>> Thermos/Scheduler
>>>>>>>>>> abstraction really hinders what type of user experience (UI) we
>> can
>>>>>>>>> build,
>>>>>>>>>> and other frameworks do a much better job by being more
>>> opinionated and
>>>>>>>>>> pulling executor data into the scheduler UI.
>>>>>>>>>>>> We probably don't need to debate on this point. Executor is a
>>> first
>>>>>>>>>> class mesos citizen and time is right for aurora to have good
>>> support for
>>>>>>>>>> it.In our case, we have kubernetes like pods modeled through
>>>>>>>>> docker-compose
>>>>>>>>>> and the executor manages that. Scheduler UI main features should
>>> not get
>>>>>>>>>> bogged down or be held back for a particular executor. That feels=

>>>>>>>>>> incredibly restrictive. If a special UI mode for thermos executor=

>>> is
>>>>>>>>>> created, that should be fine. We do have to differentiate the
>>> scheduler
>>>>>>>>> and
>>>>>>>>>> thermos and aurora team has done a great job of not hard coupling=

>>> the
>>>>>>>>> two.
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> 3. I'd like to add support for some first class Mesos task
>> related
>>>>>>>>>> fields.
>>>>>>>>>>> These would be optional and/or have defaults so that the regular=

>>> Aurora
>>>>>>>>>> CLI
>>>>>>>>>>> does not break. One of the examples I'm interested in
>> integrating
>>> is
>>>>>>>>> the
>>>>>>>>>>> Mesos Fetcher so that resources can be downloaded into the
>>> sandbox that
>>>>>>>>>> the
>>>>>>>>>>> custom executor may need. (The executor path will never be
>>> exposed as
>>>>>>>>>> this
>>>>>>>>>>> will be defined serverside and be static).
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> Wouldn't the mesos-fetcher call just be another process to be
>>> passed to
>>>>>>>>>> your executor? I have to admit I don't know enough about how the
>>> Mesos
>>>>>>>>>> fetcher works.
>>>>>>>>>>>> Apache Mesos - Fetcher
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>> We may add other first class fields.
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> If anyone has any feedback on these, I'd be very glad to hear
>> it.
>>>>>>>>>>>=20
>>>>>>>>>>> That's it for me for now, thanks!
>>>>>>>>>>>=20
>>>>>>>>>>> -Renan
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> |
>>>>>>>>>> |
>>>>>>>>>> |
>>>>>>>>>> |  |    |
>>>>>>>>>>=20
>>>>>>>>>> |
>>>>>>>>>>=20
>>>>>>>>>> |
>>>>>>>>>> |
>>>>>>>>>> |  |
>>>>>>>>>> Apache Mesos - Fetcher
>>>>>>>>>> |  |
>>>>>>>>>>=20
>>>>>>>>>> |
>>>>>>>>>>=20
>>>>>>>>>> |
>=20
>=20
>=20
>=20
> --=20
> Cheers,
>=20
> Zhitao Li