drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5957) Wire protocol versioning, version negotiation
Date Mon, 13 Nov 2017 03:53:01 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249084#comment-16249084

Paul Rogers commented on DRILL-5957:

[~tdunning], good points as usual!

The end user is really the driver of version compatibility support. Drill users have been
fairly silent on this front, so perhaps version compatibility is not necessary in this particular
case. But, based on experience with other products, folks in production often need to:

* Upgrade clients and servers on slightly different schedules, forcing the need for older
clients to talk with newer servers or visa-versa.
* If clients are installed on desktops (such as ODBC for Tableau), then the clients may actually
be remote (or on a plane or at a customer site) at the time of server upgrades, forcing the
need for older clients to connect to new servers.
* If users have multiple Drill clusters, then they may have different upgrade schedules, requiring
that a single client be able to speak with multiple server versions. (In the most obvious
case, a user may have version X in production, while trying out version (X+1) in a test environment
prior to upgrade.)

Perhaps there are alternatives in the big data world. Maybe two versions of the Drill client
can coexist in the same Tableau or other app? (For JDBC, this would mean that the clients
must be in separate name spaces, much as SQuirreL does, but SQLline does not do.)

Will there be a performance hit to "transcode" vectors across versions when the version changes?
Of course. The question is, is the temporary performance hit an acceptable cost to allow a
staged upgrade? Or, would the users prefer to do an all-at-once upgrade in order to avoid
the performance hit? (And, of course, the performance hit creates a very good incentive to

Finally, note that the "dbody" portion of of a Drill message exists outside of the Protobuf
structure. A Drill message has four parts:

* Message ID
* P-body (Protobuf body) length
* P-body (Serialized Protobuf content)
* D-body (data body) length
* D-body (serialized value vectors)

For this reason, Protobuf formats don't help us with vector serialization. Note that vector
data may be GB in size, so sending two copies is a worse performance impact than transcoding...

All this said, if staged upgrades and version compatibility is not a concern for Drill users
at present, then there is no barrier to upgrading our vector formats; we just require new
clients be used with the new Drill version. This would, of course, be the simplest solution
by far.

> Wire protocol versioning, version negotiation
> ---------------------------------------------
>                 Key: DRILL-5957
>                 URL: https://issues.apache.org/jira/browse/DRILL-5957
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.11.0
>            Reporter: Paul Rogers
> Drill has very limited support for evolving its wire protocol. As Drill becomes more
widely deployed, this limitation will constrain the project's ability to rapidly evolve the
wire protocol based on user experience to improve simplicitly, performance or minimize resource
> Proposed is a standard mechanism to version the API and negotiate the API version between
client and server at connect time. The focus here is between Drill clients (JDBC, ODBC) and
the Drill server. The same mechanism can also be used between servers to support rolling upgrades.
> This proposal is an outline; it is not a detailed design. The purpose here is to drive
understanding of the problem. Once we have that, we can focus on the implementation details.
> h4. Problem Statement
> The problem we wish to address here concerns both the _syntax_ and _semantics_ of API
messages. Syntax concerns:
> * The set of messages and their sequence
> * The format of bytes on the wire
> * The format of message packets
> Semantics concerns:
> * The meaning of each field.
> * The layout of non-message data (vectors, in Drill.)
> We wish to introduce a system whereby both syntax and semantics can be evolved in a controlled,
known manner such that:
> * A client of version x can connect to, and interoperate with, a server in a range of
versions (x-y, x+z) for some values of y and z.
> For example, version x of the Drill client is deployed in the field. It must connect
to the oldest Drill cluster available to that client. (That is it must connect to servers
up to y versions old.) During an upgrade, the server may be upgraded before the client. Thus,
the client must also work with servers up to z versions newer than the client.
> If we wish to tackle rolling upgrades, then y and z can both be 1 for server-to-server
APIs. A version x server will talk with (x-1) servers when the cluster upgrades to x, and
will talk to (x+1) servers when the cluster is upgraded to version (x+1).
> h4. Current State
> Drill currently provides some ad-hoc version compatibility:
> * Slow change. Drill's APIs have not changed much since Drill 1.0, thereby avoiding the
> * Protobuf support. Drill uses Protobuf for message bodies, leveraging that format's
ability to absorb the additional or deprecation of individual fields.
> * API version number. The API holds a version number, though the code to use it is rather
> The above has allowed clever coding to handle some version changes, but each is a one-off,
ad-hoc collision. The recent security work is an example that, with enough effort, ad-hoc
solutions can be found.
> The above cannot handle:
> * Change in the message order
> * Change in the "pbody/dbody" structure of each message.
> * Change in the structure of serialized value vectors.
> As a result, the current structure prevents any change to Drill's core mechanism, value
vectors, as there is no way or clients and servers to negotiate the vector wire format. For
example, Drill cannot adopt Arrow because a pre-Arrow client would not understand "dbody"
message parts encoded in Arrow format and visa-versa.
> h4. API Version
> The core of the proposal is to introduce an API version. This is a simple integer which
is incremented each time that a breaking change is made to the API. (If the change can be
absorbed by the Protobuf mechanism, then it is not a breaking change.) Note that the API version
*is not* the same as the product version. Two different Drill versions may have the same API
version if nothing changed in the API.
> h4. Version Negotiation
> Given a set of well-defined protocol versions, we can next define the version negotiation
protocol between client and server:
> * The client connects and sends a "hello" message that identifies the range of API versions
that it supports, with the newest version being the version of the client itself.
> * The server receives the message and computes the version of the session as the newest
client version the the server supports.
> * The server returns this version to the client which switches to the selected API version.
(The server returns an error, and disconnects, if there is no common version.)
> * The server and client use only messages valid for the given API version. This may mean
converting data from one representation to another.
> The above is pretty standard.
> h4. Backward Compatibility Implementation
> Consider a server that must work with its own version (version c) and, say, two older
versions (a and b).
> In most cases, changes across versions are minor. Perhaps version b introduced a better
error reporting format (akin to SQLWARN and SQLERROR codes). Version c may have change the
layout of a particular vector (or introduce a new vector type.) How does the server handle
these when talking with older clients?
> If a version c server talks to a version a client, then it must retain the old version
a way of reporting errors. If the same version talks to a version b or c server, it uses the
new form. This simply means that, at the point that an error response is sent to the client,
the server selects either the a version of that message or the b version.
> Since the a version previously existed, and the b version was added, it is straightforward
(if tedious) or the developer to retain both versions and use them depending on the client
> Now, consider version c that introduced a new vector format. Here, the server must transcode
the data to the a version for clients of version a or b. Again, since the a version existed,
and the c version was added, the developer is aware of both. The transcoding step is new,
and does introduce a performance hit. But, it does allow the newer format to be rewritten
in the older wire format.
> For example, perhaps version c modified offset vectors to encode the end position of
each row, rather than the start position (with the start position of the first row implied
to be at 0.) This causes the offset vector to shrink by one position. Because of power-of-two
rounding, this may halve the memory required. If the server talks to a version a or b client,
then it simply transcodes the vector by shifting it upward and inserting the required 0 value
at the zero position.
> The same logic applies to the client as well, when receiving data. A version c client,
when working with a version a server, must transcode the old offset vector format to the c
> h4. Bootstrap
> The above is fine for new Drill versions. But, how would we get started? This is the
bootstrap problem: we can't change to the new version until both clients and servers understand
the new version. But, since Drill does not currently support versioning, we can't introduce
versioning until Drill understands versioning: a catch-22 situation.
> We can observe that, initially, we are mostly concerned with client/server communication:
old clients will exist in the field even after the upgrade to the version-aware server. We
also observe that the client initiates the conversation.
> One solution is to leverage the existing client "hello" message. Suppose we introduce
the above versioning protocol in API version 10 (say). When a 9 or older version client connects,
it will send the existing Protobuf "hello" message. The version 10 server can use that to
detect the older client, and immediately switch to the old (unversioned) protocol.
> Version 10 clients introduce a new V2 "hello" message that contains the schema negotiation
information. If the server receives this, then the server knows it is working with a version
10 (or later) client, and will use that protocol instead, complete with version negotiation
and adaptation as described above.
> h4. Alternatives
> One solution is to require uses to upgrade their drivers at the same moment that they
upgrade their servers. This is often impractical, however, and is not possible if a single
driver must work with multiple Drill servers upgraded at different times.
> Suppose that we decide not to introduce versioning at the API level, but instead soldier
on with what we have. We will be able to:
> * Carefully introduce new messages and new Protobuf fields.
> We will *not* be able to:
> * Change the wire format for any non-Protobuf data, especially value vectors.
> The result is that we will be frozen at the Drill 1.0 version of the value vector format
(and so we must hope that that original format was prescient about our future needs.) We cannot
improve vector efficiency, nor can we switch to the Arrow format.
> Unfortunately, our competitors will continue to move forward with their formats, placing
Drill at a competitive disadvantage. So, in order to remain competitive, we have little alternative.

This message was sent by Atlassian JIRA

View raw message