aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renan DelValle <rdelv...@binghamton.edu>
Subject Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields
Date Sun, 12 Jun 2016 18:00:20 GMT
Hi David,

On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <david@dmclaughlin.com>
wrote:

> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
> meghdoot_b@yahoo.com.invalid> wrote:
>
> > Comments inline
> >
> >       From: David McLaughlin <dmclaughlin@apache.org>
> >  To: dev@aurora.apache.org
> >  Sent: Thursday, June 9, 2016 5:13 PM
> >  Subject: Re: Golang Aurora lib, multiple executor support, integrate
> > mesos task related fields
> >
> > On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rdelval1@binghamton.edu>
> > wrote:
> >
> > > Hello all,
> > >
> > > I'd like to (re-)introduce myself. My name's Renan DelValle and I've
> had
> > > the pleasure of being part of the Aurora community for the last year or
> > so.
> > >
> > > Last year I worked to allow Aurora to utilize custom executors. With
> the
> > > help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
> > became
> > > a reality and made into Aurora late last year. (Also got to show an
> early
> > > beta at MesosCon!)
> > >
> > > For this year's summer, I have some new goals which I invite anyone to
> > > provide input on.
> > >
> > > 1. Create a Golang library that works as an alternative to Aurora's
> > Python
> > > client and Pystachio. The initial goal of this library is to support
> the
> > > most critical job related Thrift API's. (Admin operations can continue
> to
> > > be done from the Aurora CLI). Since we support custom executors, we
> need
> > a
> > > way that's not so tied to thermos (as Pystachio is) to send jobs
> > > configurations to Aurora (and by extension the custom executor).
> > >
> >
> >
> > You can easily add support for a different configuration format without
> > having to rewrite the CLI in Go. You'd do that by adding your own custom
> > noun/verbs to the client or replacing existing commands. For example, at
> > Twitter we have our own version of 'aurora update start' that plugs into
> an
> > internal Deploy Orchestration Service and we have a whole other command
> for
> > federated deploys. I can show you how we do that.
> >
> > The first thing I thought of though was - this seems like a perfect
> > starting point to get serious about a HTTP+JSON API for Aurora. Having
> that
> > would make it trivial to do a CLI in any language you want, and the
> Python
> > CLI would really only be there to do Pystachio evaluation.
> > >> CLI integrations is not useful for a lot of different reasons. We need
> > API's from other orchestration points from different components and hence
> > we would package it in a go library. Uber, I believe has taken the same
> > route (info from this mesoscon)
> > We are not writing a replacement cli tool. As noted admin operations
> would
> > heavily leverage current aurora cli.
> > I think REST work has been postponed and attempted few times in past. We
> > cannot wait for that.
> >
>
>
> This is the dev list for contributions to the project, so I assumed we were
> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
> has Go bindings so you're all set for whatever custom tooling you're
> building. Please use the users list for feedback on that.


While I'm very thankful for your input on the matter, my original text very
clearly said:
"1. Create a Golang library that works as an *alternative* to
Aurora's Python client and Pystachio."

I believe every single item I have listed has the potential to become a
useful contribution to this project.
Therefore, I feel that it is prudent continue the discussion here (unless
others feel similarly).

-Renan

>
> >
> >
> > >
> > > 2. Create support for using multiple executors within a single Aurora
> > > scheduler instance. As of right now, only a single executor can exist
> for
> > > each Aurora scheduler instance. The idea is to allow the Aurora
> scheduler
> > > to support multiple executors such that they can be used alongside one
> > > another, providing flexibility for running jobs.
> > >
> >
> > Just for my own understanding - what problems are you solving with your
> > custom executor? Sorry if you've explained this to the lists before.
> >
> > I ask because I'd really like to see Aurora stop treating the executor
> > config as an opaque blob and being more opinionated. The
> Thermos/Scheduler
> > abstraction really hinders what type of user experience (UI) we can
> build,
> > and other frameworks do a much better job by being more opinionated and
> > pulling executor data into the scheduler UI.
> > >> We probably don't need to debate on this point. Executor is a first
> > class mesos citizen and time is right for aurora to have good support for
> > it.In our case, we have kubernetes like pods modeled through
> docker-compose
> > and the executor manages that. Scheduler UI main features should not get
> > bogged down or be held back for a particular executor. That feels
> > incredibly restrictive. If a special UI mode for thermos executor is
> > created, that should be fine. We do have to differentiate the scheduler
> and
> > thermos and aurora team has done a great job of not hard coupling the
> two.
>
>
> > >
> > > 3. I'd like to add support for some first class Mesos task related
> > fields.
> > > These would be optional and/or have defaults so that the regular Aurora
> > CLI
> > > does not break. One of the examples I'm interested in integrating is
> the
> > > Mesos Fetcher so that resources can be downloaded into the sandbox that
> > the
> > > custom executor may need. (The executor path will never be exposed as
> > this
> > > will be defined serverside and be static).
> >
> >
> > Wouldn't the mesos-fetcher call just be another process to be passed to
> > your executor? I have to admit I don't know enough about how the Mesos
> > fetcher works.
> > >> Apache Mesos - Fetcher
>
>
>
> > We may add other first class fields.
> >
> >
> > >
> > > If anyone has any feedback on these, I'd be very glad to hear it.
> > >
> > > That's it for me for now, thanks!
> > >
> > > -Renan
> > >
> >
> >
> >
> >
> > |
> > |
> > |
> > |   |    |
> >
> >    |
> >
> >   |
> > |
> > |   |
> > Apache Mesos - Fetcher
> >    |   |
> >
> >   |
> >
> >   |
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message