aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Khutornenko <ma...@apache.org>
Subject Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields
Date Mon, 13 Jun 2016 16:19:55 GMT
I am with Rick and David on this. The (1) and (2) feel like building a
parallel universe within Aurora without a clear justification on the
long-term benefits for the community.

I am strong -1 on bringing yet another language into the Aurora
ecosystem without identifying the strategic upside. As David
suggested, any efforts towards building the first-class REST API
instead would go a long way and will be much appreciated by everyone.

I also don't see a reason to support multiple executors per task. That
feels like re-creating thermos in a thermos-less world.

As for (3), I'd like to see more details on the mechanics here but
generally positive towards supporting more TaskInfo features in
Aurora.


On Sun, Jun 12, 2016 at 11:46 AM,  <rick@chartbeat.com> wrote:
> I generally shy away from technical goals that are based on a choice of language. What
is it about a go client that can't be done by extending the existing client?
>
>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rdelval1@binghamton.edu> wrote:
>>
>> Hi David,
>>
>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <david@dmclaughlin.com>
>> wrote:
>>
>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>>> meghdoot_b@yahoo.com.invalid> wrote:
>>>
>>>> Comments inline
>>>>
>>>>      From: David McLaughlin <dmclaughlin@apache.org>
>>>> To: dev@aurora.apache.org
>>>> Sent: Thursday, June 9, 2016 5:13 PM
>>>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>>>> mesos task related fields
>>>>
>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rdelval1@binghamton.edu>
>>>> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've
>>> had
>>>>> the pleasure of being part of the Aurora community for the last year
or
>>>> so.
>>>>>
>>>>> Last year I worked to allow Aurora to utilize custom executors. With
>>> the
>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
>>>> became
>>>>> a reality and made into Aurora late last year. (Also got to show an
>>> early
>>>>> beta at MesosCon!)
>>>>>
>>>>> For this year's summer, I have some new goals which I invite anyone to
>>>>> provide input on.
>>>>>
>>>>> 1. Create a Golang library that works as an alternative to Aurora's
>>>> Python
>>>>> client and Pystachio. The initial goal of this library is to support
>>> the
>>>>> most critical job related Thrift API's. (Admin operations can continue
>>> to
>>>>> be done from the Aurora CLI). Since we support custom executors, we
>>> need
>>>> a
>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>>> configurations to Aurora (and by extension the custom executor).
>>>>
>>>>
>>>> You can easily add support for a different configuration format without
>>>> having to rewrite the CLI in Go. You'd do that by adding your own custom
>>>> noun/verbs to the client or replacing existing commands. For example, at
>>>> Twitter we have our own version of 'aurora update start' that plugs into
>>> an
>>>> internal Deploy Orchestration Service and we have a whole other command
>>> for
>>>> federated deploys. I can show you how we do that.
>>>>
>>>> The first thing I thought of though was - this seems like a perfect
>>>> starting point to get serious about a HTTP+JSON API for Aurora. Having
>>> that
>>>> would make it trivial to do a CLI in any language you want, and the
>>> Python
>>>> CLI would really only be there to do Pystachio evaluation.
>>>>>> CLI integrations is not useful for a lot of different reasons. We
need
>>>> API's from other orchestration points from different components and hence
>>>> we would package it in a go library. Uber, I believe has taken the same
>>>> route (info from this mesoscon)
>>>> We are not writing a replacement cli tool. As noted admin operations
>>> would
>>>> heavily leverage current aurora cli.
>>>> I think REST work has been postponed and attempted few times in past. We
>>>> cannot wait for that.
>>>
>>>
>>> This is the dev list for contributions to the project, so I assumed we were
>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
>>> has Go bindings so you're all set for whatever custom tooling you're
>>> building. Please use the users list for feedback on that.
>>
>>
>> While I'm very thankful for your input on the matter, my original text very
>> clearly said:
>> "1. Create a Golang library that works as an *alternative* to
>> Aurora's Python client and Pystachio."
>>
>> I believe every single item I have listed has the potential to become a
>> useful contribution to this project.
>> Therefore, I feel that it is prudent continue the discussion here (unless
>> others feel similarly).
>>
>> -Renan
>>
>>>
>>>>
>>>>
>>>>>
>>>>> 2. Create support for using multiple executors within a single Aurora
>>>>> scheduler instance. As of right now, only a single executor can exist
>>> for
>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>>> scheduler
>>>>> to support multiple executors such that they can be used alongside one
>>>>> another, providing flexibility for running jobs.
>>>>
>>>> Just for my own understanding - what problems are you solving with your
>>>> custom executor? Sorry if you've explained this to the lists before.
>>>>
>>>> I ask because I'd really like to see Aurora stop treating the executor
>>>> config as an opaque blob and being more opinionated. The
>>> Thermos/Scheduler
>>>> abstraction really hinders what type of user experience (UI) we can
>>> build,
>>>> and other frameworks do a much better job by being more opinionated and
>>>> pulling executor data into the scheduler UI.
>>>>>> We probably don't need to debate on this point. Executor is a first
>>>> class mesos citizen and time is right for aurora to have good support for
>>>> it.In our case, we have kubernetes like pods modeled through
>>> docker-compose
>>>> and the executor manages that. Scheduler UI main features should not get
>>>> bogged down or be held back for a particular executor. That feels
>>>> incredibly restrictive. If a special UI mode for thermos executor is
>>>> created, that should be fine. We do have to differentiate the scheduler
>>> and
>>>> thermos and aurora team has done a great job of not hard coupling the
>>> two.
>>>
>>>
>>>>>
>>>>> 3. I'd like to add support for some first class Mesos task related
>>>> fields.
>>>>> These would be optional and/or have defaults so that the regular Aurora
>>>> CLI
>>>>> does not break. One of the examples I'm interested in integrating is
>>> the
>>>>> Mesos Fetcher so that resources can be downloaded into the sandbox that
>>>> the
>>>>> custom executor may need. (The executor path will never be exposed as
>>>> this
>>>>> will be defined serverside and be static).
>>>>
>>>>
>>>> Wouldn't the mesos-fetcher call just be another process to be passed to
>>>> your executor? I have to admit I don't know enough about how the Mesos
>>>> fetcher works.
>>>>>> Apache Mesos - Fetcher
>>>
>>>
>>>
>>>> We may add other first class fields.
>>>>
>>>>
>>>>>
>>>>> If anyone has any feedback on these, I'd be very glad to hear it.
>>>>>
>>>>> That's it for me for now, thanks!
>>>>>
>>>>> -Renan
>>>>
>>>>
>>>>
>>>>
>>>> |
>>>> |
>>>> |
>>>> |   |    |
>>>>
>>>>   |
>>>>
>>>>  |
>>>> |
>>>> |   |
>>>> Apache Mesos - Fetcher
>>>>   |   |
>>>>
>>>>  |
>>>>
>>>>  |
>>>

Mime
View raw message