aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r...@chartbeat.com
Subject Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields
Date Sun, 12 Jun 2016 18:46:27 GMT
I generally shy away from technical goals that are based on a choice of language. What is it
about a go client that can't be done by extending the existing client?

> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rdelval1@binghamton.edu> wrote:
> 
> Hi David,
> 
> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <david@dmclaughlin.com>
> wrote:
> 
>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>> meghdoot_b@yahoo.com.invalid> wrote:
>> 
>>> Comments inline
>>> 
>>>      From: David McLaughlin <dmclaughlin@apache.org>
>>> To: dev@aurora.apache.org
>>> Sent: Thursday, June 9, 2016 5:13 PM
>>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>>> mesos task related fields
>>> 
>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rdelval1@binghamton.edu>
>>> wrote:
>>> 
>>>> Hello all,
>>>> 
>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've
>> had
>>>> the pleasure of being part of the Aurora community for the last year or
>>> so.
>>>> 
>>>> Last year I worked to allow Aurora to utilize custom executors. With
>> the
>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
>>> became
>>>> a reality and made into Aurora late last year. (Also got to show an
>> early
>>>> beta at MesosCon!)
>>>> 
>>>> For this year's summer, I have some new goals which I invite anyone to
>>>> provide input on.
>>>> 
>>>> 1. Create a Golang library that works as an alternative to Aurora's
>>> Python
>>>> client and Pystachio. The initial goal of this library is to support
>> the
>>>> most critical job related Thrift API's. (Admin operations can continue
>> to
>>>> be done from the Aurora CLI). Since we support custom executors, we
>> need
>>> a
>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>> configurations to Aurora (and by extension the custom executor).
>>> 
>>> 
>>> You can easily add support for a different configuration format without
>>> having to rewrite the CLI in Go. You'd do that by adding your own custom
>>> noun/verbs to the client or replacing existing commands. For example, at
>>> Twitter we have our own version of 'aurora update start' that plugs into
>> an
>>> internal Deploy Orchestration Service and we have a whole other command
>> for
>>> federated deploys. I can show you how we do that.
>>> 
>>> The first thing I thought of though was - this seems like a perfect
>>> starting point to get serious about a HTTP+JSON API for Aurora. Having
>> that
>>> would make it trivial to do a CLI in any language you want, and the
>> Python
>>> CLI would really only be there to do Pystachio evaluation.
>>>>> CLI integrations is not useful for a lot of different reasons. We need
>>> API's from other orchestration points from different components and hence
>>> we would package it in a go library. Uber, I believe has taken the same
>>> route (info from this mesoscon)
>>> We are not writing a replacement cli tool. As noted admin operations
>> would
>>> heavily leverage current aurora cli.
>>> I think REST work has been postponed and attempted few times in past. We
>>> cannot wait for that.
>> 
>> 
>> This is the dev list for contributions to the project, so I assumed we were
>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
>> has Go bindings so you're all set for whatever custom tooling you're
>> building. Please use the users list for feedback on that.
> 
> 
> While I'm very thankful for your input on the matter, my original text very
> clearly said:
> "1. Create a Golang library that works as an *alternative* to
> Aurora's Python client and Pystachio."
> 
> I believe every single item I have listed has the potential to become a
> useful contribution to this project.
> Therefore, I feel that it is prudent continue the discussion here (unless
> others feel similarly).
> 
> -Renan
> 
>> 
>>> 
>>> 
>>>> 
>>>> 2. Create support for using multiple executors within a single Aurora
>>>> scheduler instance. As of right now, only a single executor can exist
>> for
>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>> scheduler
>>>> to support multiple executors such that they can be used alongside one
>>>> another, providing flexibility for running jobs.
>>> 
>>> Just for my own understanding - what problems are you solving with your
>>> custom executor? Sorry if you've explained this to the lists before.
>>> 
>>> I ask because I'd really like to see Aurora stop treating the executor
>>> config as an opaque blob and being more opinionated. The
>> Thermos/Scheduler
>>> abstraction really hinders what type of user experience (UI) we can
>> build,
>>> and other frameworks do a much better job by being more opinionated and
>>> pulling executor data into the scheduler UI.
>>>>> We probably don't need to debate on this point. Executor is a first
>>> class mesos citizen and time is right for aurora to have good support for
>>> it.In our case, we have kubernetes like pods modeled through
>> docker-compose
>>> and the executor manages that. Scheduler UI main features should not get
>>> bogged down or be held back for a particular executor. That feels
>>> incredibly restrictive. If a special UI mode for thermos executor is
>>> created, that should be fine. We do have to differentiate the scheduler
>> and
>>> thermos and aurora team has done a great job of not hard coupling the
>> two.
>> 
>> 
>>>> 
>>>> 3. I'd like to add support for some first class Mesos task related
>>> fields.
>>>> These would be optional and/or have defaults so that the regular Aurora
>>> CLI
>>>> does not break. One of the examples I'm interested in integrating is
>> the
>>>> Mesos Fetcher so that resources can be downloaded into the sandbox that
>>> the
>>>> custom executor may need. (The executor path will never be exposed as
>>> this
>>>> will be defined serverside and be static).
>>> 
>>> 
>>> Wouldn't the mesos-fetcher call just be another process to be passed to
>>> your executor? I have to admit I don't know enough about how the Mesos
>>> fetcher works.
>>>>> Apache Mesos - Fetcher
>> 
>> 
>> 
>>> We may add other first class fields.
>>> 
>>> 
>>>> 
>>>> If anyone has any feedback on these, I'd be very glad to hear it.
>>>> 
>>>> That's it for me for now, thanks!
>>>> 
>>>> -Renan
>>> 
>>> 
>>> 
>>> 
>>> |
>>> |
>>> |
>>> |   |    |
>>> 
>>>   |
>>> 
>>>  |
>>> |
>>> |   |
>>> Apache Mesos - Fetcher
>>>   |   |
>>> 
>>>  |
>>> 
>>>  |
>> 

Mime
View raw message