aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Sirois <john.sir...@gmail.com>
Subject [RFC] REST / thrift & AURORA-987
Date Mon, 30 Nov 2015 20:16:50 GMT
I’ve experimenting on https://issues.apache.org/jira/browse/AURORA-987 for
the past few weeks and I’d like to ask for feedback on the direction I’d
like to head. If you’re interested in the evolution of the Aurora REST api,
read on.
------------------------------

AURORA-987 aims to create a first-class REST-like scheduler interface. I’ve
re-familiarized myself with the codebase and come to the conclusion that
transitioning to a 1st class REST api requires maintaining the core thrift
API as the 1st class API until the point the REST API is fully established
and clients can all be transitioned.

I think this conclusion is probably uncontroversial, but the key factors
pushing this way are:

   1.

   The thrift API has both wide and deep dependencies inside the Aurora
   codebase - 276 imports across 97 files:

   $ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
-v "import org.apache.aurora.gen.storage" | wc -l
   276
   $ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
-v "import org.apache.aurora.gen.storage" | cut -d: -f1 | sort -u | wc
-l
   97

   2.

   The thrift API is stored long-term in the log in serialized form.

Both 1 & 2 dictate that the thrift API, at least its enums, structs and
unions, must be maintained for the forseeable future.
We also have the RPC API (thrift services) - which is currently a ~thin,
but not insignificant, container of API processing logic. For example, see
here
<https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java#L220-L267>
.

As such it seems to me the REST API should call into the existing thrift
API to provide a stable transition and confidence in core logic of API
method implementations.

This leads to the following ideas for paths forward:

   1. Hand construct a REST forwarding layer and maintain it in tandem with
   thrift API changes.
   2. Automate 1 such that thrift API changes cause REST API changes
   automatically.

The hand construction path has the obvious maintenance issues, but is
otherwise straight-forward. The maintenance issues should not be
overstated, since good tests and some extra review vigilance could be
enough to make the approach work for the period of time both APIs are
supported.

That said, an automated solution with a single source of truth for the API
definition is clearly preferrable given the automation is free.
The automation is far from free though and so I’ve started investigating
one approach to this automation to flesh out the scope.

We already do our own thrift codegen
<https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py>
via a custom gradle ThriftEntitiesPlugin
<https://github.com/apache/aurora/blob/master/buildSrc/src/main/groovy/org/apache/aurora/build/ThriftEntitiesPlugin.groovy>
that works around Apache thrift’s java codegen in order to generate
immutable wrapper entities for the storage system.
I propose taking this further and generating our own thrift API and
entities in 1 pass through our thrift files. These would be immutable
thrift entities 1st class with builders for modification and the entities
and the generated service interfaces would carry extra metadata in the form
of annotations to bind REST services and their metadata with.

There are 2 paths I’ve considered towards this end:

   1. Modify Apache thrift to support immutable-style java output with
   support for thrift annotations.
   2. Write our own thrift parser and code generator to do said same.

I’ve been pursuing option 2 even though it sounds worse on its face. The
swift <https://github.com/facebook/swift> project from Facebook brings
options 1 and 2 back on the same level of undertaking since the parsing and
protocol implementations can be leveraged as libraries and only the codegen
portion need be undertaken (You can see some of that work here
<https://github.com/apache/aurora/compare/master...jsirois:jsirois/issues/AURORA-987/experiments>
).

So, with that background 2 questions of the same form:

   1. Is there some other fundamental approach I’m missing to bolting on a
   1st class REST API, or is the hand construction approach favorable?
   2. Is the approach to single point of API control using swift misguided?
   Should I be focusing on Apache thrift enhancement instead? Should I be
   generating the *.thrift files instead from a new 1st class source of truth
   in the form of a json api schema?

Any and all feedback is welcome!
​

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message