aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Morozov <igm...@gmail.com>
Subject Re: [RFC] REST / thrift & AURORA-987
Date Tue, 01 Dec 2015 18:17:26 GMT
We had very similar concerns here at Uber in regards of Thrift and
newer REST API that is coming to Aurora scheduler. It feels like
having a general API model in a scheduler and providing whatever
interface is necessary/convenient for integration would generally be a
better option then building REST API layer on top of Thrift API.


On Tue, Dec 1, 2015 at 9:37 AM, David McLaughlin <dmclaughlin@apache.org> wrote:
> Shouldn't we start with the design of the API itself? Won't that influence
> many of the answers to these questions?
>
> E.g. if you're just looking to port the Thrift API 1:1 to a JSON + HTTP
> interface then that's a very different set of requirements to starting
> fresh and doing a better job with our API.
>
> Personally I don't think the existing Thrift API is a very good base to
> build an API on top off. A lot of the endpoints are fit for one purpose
> (e.g. a specific UI view or client function) rather than being flexible. I
> can't tell you how many times we wanted to go in and improve the UI in some
> way only to find the existing API does not give us access to the data we
> want.
>
> So yeah, I feel like the API should be more generic with regards to data
> access. So fewer, more-powerful endpoints that support complex queries.
>
> On Mon, Nov 30, 2015 at 12:16 PM, John Sirois <john.sirois@gmail.com> wrote:
>
>> I’ve experimenting on https://issues.apache.org/jira/browse/AURORA-987 for
>> the past few weeks and I’d like to ask for feedback on the direction I’d
>> like to head. If you’re interested in the evolution of the Aurora REST api,
>> read on.
>> ------------------------------
>>
>> AURORA-987 aims to create a first-class REST-like scheduler interface. I’ve
>> re-familiarized myself with the codebase and come to the conclusion that
>> transitioning to a 1st class REST api requires maintaining the core thrift
>> API as the 1st class API until the point the REST API is fully established
>> and clients can all be transitioned.
>>
>> I think this conclusion is probably uncontroversial, but the key factors
>> pushing this way are:
>>
>>    1.
>>
>>    The thrift API has both wide and deep dependencies inside the Aurora
>>    codebase - 276 imports across 97 files:
>>
>>    $ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
>> -v "import org.apache.aurora.gen.storage" | wc -l
>>    276
>>    $ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
>> -v "import org.apache.aurora.gen.storage" | cut -d: -f1 | sort -u | wc
>> -l
>>    97
>>
>>    2.
>>
>>    The thrift API is stored long-term in the log in serialized form.
>>
>> Both 1 & 2 dictate that the thrift API, at least its enums, structs and
>> unions, must be maintained for the forseeable future.
>> We also have the RPC API (thrift services) - which is currently a ~thin,
>> but not insignificant, container of API processing logic. For example, see
>> here
>> <
>> https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java#L220-L267
>> >
>> .
>>
>> As such it seems to me the REST API should call into the existing thrift
>> API to provide a stable transition and confidence in core logic of API
>> method implementations.
>>
>> This leads to the following ideas for paths forward:
>>
>>    1. Hand construct a REST forwarding layer and maintain it in tandem with
>>    thrift API changes.
>>    2. Automate 1 such that thrift API changes cause REST API changes
>>    automatically.
>>
>> The hand construction path has the obvious maintenance issues, but is
>> otherwise straight-forward. The maintenance issues should not be
>> overstated, since good tests and some extra review vigilance could be
>> enough to make the approach work for the period of time both APIs are
>> supported.
>>
>> That said, an automated solution with a single source of truth for the API
>> definition is clearly preferrable given the automation is free.
>> The automation is far from free though and so I’ve started investigating
>> one approach to this automation to flesh out the scope.
>>
>> We already do our own thrift codegen
>> <
>> https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py
>> >
>> via a custom gradle ThriftEntitiesPlugin
>> <
>> https://github.com/apache/aurora/blob/master/buildSrc/src/main/groovy/org/apache/aurora/build/ThriftEntitiesPlugin.groovy
>> >
>> that works around Apache thrift’s java codegen in order to generate
>> immutable wrapper entities for the storage system.
>> I propose taking this further and generating our own thrift API and
>> entities in 1 pass through our thrift files. These would be immutable
>> thrift entities 1st class with builders for modification and the entities
>> and the generated service interfaces would carry extra metadata in the form
>> of annotations to bind REST services and their metadata with.
>>
>> There are 2 paths I’ve considered towards this end:
>>
>>    1. Modify Apache thrift to support immutable-style java output with
>>    support for thrift annotations.
>>    2. Write our own thrift parser and code generator to do said same.
>>
>> I’ve been pursuing option 2 even though it sounds worse on its face. The
>> swift <https://github.com/facebook/swift> project from Facebook brings
>> options 1 and 2 back on the same level of undertaking since the parsing and
>> protocol implementations can be leveraged as libraries and only the codegen
>> portion need be undertaken (You can see some of that work here
>> <
>> https://github.com/apache/aurora/compare/master...jsirois:jsirois/issues/AURORA-987/experiments
>> >
>> ).
>>
>> So, with that background 2 questions of the same form:
>>
>>    1. Is there some other fundamental approach I’m missing to bolting on a
>>    1st class REST API, or is the hand construction approach favorable?
>>    2. Is the approach to single point of API control using swift misguided?
>>    Should I be focusing on Apache thrift enhancement instead? Should I be
>>    generating the *.thrift files instead from a new 1st class source of
>> truth
>>    in the form of a json api schema?
>>
>> Any and all feedback is welcome!
>>
>>



-- 
-Igor

Mime
View raw message