aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From meghdoot bhattacharya <>
Subject Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields
Date Fri, 10 Jun 2016 00:44:22 GMT
Comments inline

      From: David McLaughlin <>
 Sent: Thursday, June 9, 2016 5:13 PM
 Subject: Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields
On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <>

> Hello all,
> I'd like to (re-)introduce myself. My name's Renan DelValle and I've had
> the pleasure of being part of the Aurora community for the last year or so.
> Last year I worked to allow Aurora to utilize custom executors. With the
> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature became
> a reality and made into Aurora late last year. (Also got to show an early
> beta at MesosCon!)
> For this year's summer, I have some new goals which I invite anyone to
> provide input on.
> 1. Create a Golang library that works as an alternative to Aurora's Python
> client and Pystachio. The initial goal of this library is to support the
> most critical job related Thrift API's. (Admin operations can continue to
> be done from the Aurora CLI). Since we support custom executors, we need a
> way that's not so tied to thermos (as Pystachio is) to send jobs
> configurations to Aurora (and by extension the custom executor).

You can easily add support for a different configuration format without
having to rewrite the CLI in Go. You'd do that by adding your own custom
noun/verbs to the client or replacing existing commands. For example, at
Twitter we have our own version of 'aurora update start' that plugs into an
internal Deploy Orchestration Service and we have a whole other command for
federated deploys. I can show you how we do that.

The first thing I thought of though was - this seems like a perfect
starting point to get serious about a HTTP+JSON API for Aurora. Having that
would make it trivial to do a CLI in any language you want, and the Python
CLI would really only be there to do Pystachio evaluation.
>> CLI integrations is not useful for a lot of different reasons. We need API's from
other orchestration points from different components and hence we would package it in a go
library. Uber, I believe has taken the same route (info from this mesoscon)
We are not writing a replacement cli tool. As noted admin operations would heavily leverage
current aurora cli.
I think REST work has been postponed and attempted few times in past. We cannot wait for that.

> 2. Create support for using multiple executors within a single Aurora
> scheduler instance. As of right now, only a single executor can exist for
> each Aurora scheduler instance. The idea is to allow the Aurora scheduler
> to support multiple executors such that they can be used alongside one
> another, providing flexibility for running jobs.

Just for my own understanding - what problems are you solving with your
custom executor? Sorry if you've explained this to the lists before.

I ask because I'd really like to see Aurora stop treating the executor
config as an opaque blob and being more opinionated. The Thermos/Scheduler
abstraction really hinders what type of user experience (UI) we can build,
and other frameworks do a much better job by being more opinionated and
pulling executor data into the scheduler UI.
>> We probably don't need to debate on this point. Executor is a first class mesos citizen
and time is right for aurora to have good support for it.In our case, we have kubernetes like
pods modeled through docker-compose and the executor manages that. Scheduler UI main features
should not get bogged down or be held back for a particular executor. That feels incredibly
restrictive. If a special UI mode for thermos executor is created, that should be fine. We
do have to differentiate the scheduler and thermos and aurora team has done a great job of
not hard coupling the two.

> 3. I'd like to add support for some first class Mesos task related fields.
> These would be optional and/or have defaults so that the regular Aurora CLI
> does not break. One of the examples I'm interested in integrating is the
> Mesos Fetcher so that resources can be downloaded into the sandbox that the
> custom executor may need. (The executor path will never be exposed as this
> will be defined serverside and be static).

Wouldn't the mesos-fetcher call just be another process to be passed to
your executor? I have to admit I don't know enough about how the Mesos
fetcher works.
>> Apache Mesos - Fetcher

We may add other first class fields.

> If anyone has any feedback on these, I'd be very glad to hear it.
> That's it for me for now, thanks!
> -Renan

|   |    |


|   |  
Apache Mesos - Fetcher
   |   |



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message