aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From meghdoot bhattacharya <>
Subject Re: Aurora-1288 custom executor support in client
Date Mon, 27 Jul 2015 07:34:51 GMT
Is the REST API interface planned to be be identical/close to the thrift api interface?
And if we leave the client dsl modifications for custom executors, for now we can build a
generic client to use the thrift api directly for supporting custom executors. Cursory look
of the client api code looks like the scheduler proxy object is calling the scheduler directly
but would be nice to have the thick client logic completely removed. The old style client
orchestrated updater code is very much present. And for example, is AcquireLock scheduler
method still used? Looks like new update method does not require it but the old update method
is the only one that needed it. But if latter is deprecated, why not remove the scheduler
method? On the CreateJob, is a lock object required? It does not seem necessary. It would
be good to document these, so a generic client can be written. Maybe that is what this ticket
should do. 
      From: Bill Farner <>
 To: "" <>; meghdoot bhattacharya <>

 Sent: Tuesday, July 21, 2015 9:53 PM
 Subject: Re: Aurora-1288 custom executor support in client
One of the biggest challenges i foresee if we embark on making the client
support custom executors is schema handling.  By design, pystachio defines
schemas that configurations must match, which will likely make it difficult
to generalize to support swappable/arbitrary executors.  I am not familiar
enough with pystachio to articulate how [in]feasible this is, so hopefully
someone else can expound on that.

While i think this is possible, i wouldn't be disappointed if this use case
were used to apply pressure to getting a REST API in place, as i believe
that's one of the few major hurdles left necessitating a CLI.


On Tue, Jul 21, 2015 at 2:42 PM, meghdoot bhattacharya <> wrote:

> As the server side changes are getting wrapped up, Bill and I were
> discussing offline around custom executor support in aurora client, one of
> the sub tasks in the ticket. So, we are bringing the discussion to the
> community to get feedback from you all.
> Meghdoot>Theidea should be to support the current DSL (to not break
> current integrations)but also introduce a new way of defining things where
> even thermos or anycustom executor can have its own stuff defined in a
> generic way eventuallymarshalled into “data” blob supported as Brian had
> commented in the ticket
> Bill> Just so i follow - what drives the requirement ofdoing this in the
> existing client?  Our current direction is to make theclient thinner over
> time, so hypothetically the surface area for a custom clientis getting
> pretty small.  I'd also like to invest in a REST API quitesoon, which
> reduces the technology coupling as well.  Do these detailschange the
> equation for you?
> Meghdoot>Generallyhaving a cli is powerful. Though we can rely on thrift
> api’s and future restapi’s directly to plug in with our web systems, having
> a cli support is apowerful functionality to have especially if we can make
> it generic to dealwith custom executors. And we are thinking of upgrading
> the aurora client andnot have some other cli.
> Inthe end any client should just send the job config and executor data
> buried inthe current field, but having a DSL where you can declare things
> can still bebeneficial.
>  Butif we really are moving in a direction that the existing cli gets used
> mostlyfor admin usage and we retire the DSL usage, then there is no point
> investingon it. Otherwise the question remains why DSL support only for
> thermos.
> Otherpoint to add if we really want to keep the DSL only for thermos, then
> to havecustom executors thrive through clients of their own, we need a
> detailed writeup on how that will work. Meaning to create or update or some
> other job relatedoperations what are the sequence of API’s to call (for
> example grab lock api,call main api, call release api), if behind the
> scenes the existing aurora client isdoing it today. Otherwisecustom
> executors will struggle in the absence of support from aurora client andan
> integration guide.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message