aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Morozov <>
Subject Re: [RFC] REST / thrift & AURORA-987
Date Tue, 01 Dec 2015 18:17:26 GMT
We had very similar concerns here at Uber in regards of Thrift and
newer REST API that is coming to Aurora scheduler. It feels like
having a general API model in a scheduler and providing whatever
interface is necessary/convenient for integration would generally be a
better option then building REST API layer on top of Thrift API.

On Tue, Dec 1, 2015 at 9:37 AM, David McLaughlin <> wrote:
> Shouldn't we start with the design of the API itself? Won't that influence
> many of the answers to these questions?
> E.g. if you're just looking to port the Thrift API 1:1 to a JSON + HTTP
> interface then that's a very different set of requirements to starting
> fresh and doing a better job with our API.
> Personally I don't think the existing Thrift API is a very good base to
> build an API on top off. A lot of the endpoints are fit for one purpose
> (e.g. a specific UI view or client function) rather than being flexible. I
> can't tell you how many times we wanted to go in and improve the UI in some
> way only to find the existing API does not give us access to the data we
> want.
> So yeah, I feel like the API should be more generic with regards to data
> access. So fewer, more-powerful endpoints that support complex queries.
> On Mon, Nov 30, 2015 at 12:16 PM, John Sirois <> wrote:
>> I’ve experimenting on for
>> the past few weeks and I’d like to ask for feedback on the direction I’d
>> like to head. If you’re interested in the evolution of the Aurora REST api,
>> read on.
>> ------------------------------
>> AURORA-987 aims to create a first-class REST-like scheduler interface. I’ve
>> re-familiarized myself with the codebase and come to the conclusion that
>> transitioning to a 1st class REST api requires maintaining the core thrift
>> API as the 1st class API until the point the REST API is fully established
>> and clients can all be transitioned.
>> I think this conclusion is probably uncontroversial, but the key factors
>> pushing this way are:
>>    1.
>>    The thrift API has both wide and deep dependencies inside the Aurora
>>    codebase - 276 imports across 97 files:
>>    $ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
>> -v "import" | wc -l
>>    276
>>    $ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
>> -v "import" | cut -d: -f1 | sort -u | wc
>> -l
>>    97
>>    2.
>>    The thrift API is stored long-term in the log in serialized form.
>> Both 1 & 2 dictate that the thrift API, at least its enums, structs and
>> unions, must be maintained for the forseeable future.
>> We also have the RPC API (thrift services) - which is currently a ~thin,
>> but not insignificant, container of API processing logic. For example, see
>> here
>> <
>> >
>> .
>> As such it seems to me the REST API should call into the existing thrift
>> API to provide a stable transition and confidence in core logic of API
>> method implementations.
>> This leads to the following ideas for paths forward:
>>    1. Hand construct a REST forwarding layer and maintain it in tandem with
>>    thrift API changes.
>>    2. Automate 1 such that thrift API changes cause REST API changes
>>    automatically.
>> The hand construction path has the obvious maintenance issues, but is
>> otherwise straight-forward. The maintenance issues should not be
>> overstated, since good tests and some extra review vigilance could be
>> enough to make the approach work for the period of time both APIs are
>> supported.
>> That said, an automated solution with a single source of truth for the API
>> definition is clearly preferrable given the automation is free.
>> The automation is far from free though and so I’ve started investigating
>> one approach to this automation to flesh out the scope.
>> We already do our own thrift codegen
>> <
>> >
>> via a custom gradle ThriftEntitiesPlugin
>> <
>> >
>> that works around Apache thrift’s java codegen in order to generate
>> immutable wrapper entities for the storage system.
>> I propose taking this further and generating our own thrift API and
>> entities in 1 pass through our thrift files. These would be immutable
>> thrift entities 1st class with builders for modification and the entities
>> and the generated service interfaces would carry extra metadata in the form
>> of annotations to bind REST services and their metadata with.
>> There are 2 paths I’ve considered towards this end:
>>    1. Modify Apache thrift to support immutable-style java output with
>>    support for thrift annotations.
>>    2. Write our own thrift parser and code generator to do said same.
>> I’ve been pursuing option 2 even though it sounds worse on its face. The
>> swift <> project from Facebook brings
>> options 1 and 2 back on the same level of undertaking since the parsing and
>> protocol implementations can be leveraged as libraries and only the codegen
>> portion need be undertaken (You can see some of that work here
>> <
>> >
>> ).
>> So, with that background 2 questions of the same form:
>>    1. Is there some other fundamental approach I’m missing to bolting on a
>>    1st class REST API, or is the hand construction approach favorable?
>>    2. Is the approach to single point of API control using swift misguided?
>>    Should I be focusing on Apache thrift enhancement instead? Should I be
>>    generating the *.thrift files instead from a new 1st class source of
>> truth
>>    in the form of a json api schema?
>> Any and all feedback is welcome!


View raw message