arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@apache.org>
Subject Re: [Discuss][Format] Arrow Flight URI scheme proposal
Date Mon, 01 Apr 2019 23:28:49 GMT
My thinking is ideally the protocol would be more opaque than engineer-y in
that an upgrade would happen as part of the negotiation process. For
example, when a connection is made, client says "hey, I also support these
things" and then server responds and says "hey, let's send data on this
channel instead". That way, end consumers don't have to worry about the
details of alternative capabilities. (E.g. this is less about a Flight
endpoint).

I like the protocol proposal with one small modification: I'd like us to
define a default protocol if one is not defined. I think it should probably
be grpc+tls but am open to other options. What do others think about a
default protocol?



On Thu, Mar 28, 2019 at 9:29 PM Wes McKinney <wesmckinn@gmail.com> wrote:

> hi David,
>
> This seems like a reasonable evolution from where we are now. I will
> defer to others to comment on the low-level details
>
> This is sort of scope and kind of a can of worms, but one area where
> we should invest some thought is alternative FlightData transports,
> while allowing the "command layer" to continue to be gRPC. One such
> possible alternative scheme includes:
>
> * gRPC-over-TCP commands (actions, etc.)
> * Movement of IPC messages using RDMA (I have never actually used RDMA
> but it has been brought up to me as a topic of interest by multiple
> parties now)
>
> If a server supports an alternative protocols (e.g. gRPC-based for
> compatibility, plus RDMA for clients that implement it) then perhaps
> this information can be encoded in URIs. I'm not sure if there's prior
> art to look at on this design-wise
>
> - Wes
>
> On Wed, Mar 27, 2019 at 1:24 PM David Li <li.davidm96@gmail.com> wrote:
> >
> > Hi all,
> >
> > We'd like to propose a URI scheme for Flight, in anticipation of
> > supporting multiple transports, and different configurations of the
> > gRPC transport. This will change Flight.proto[1] in format/ in a
> > backwards-incompatible way. This aims to fix ARROW-4651[2].
> >
> > The proposal can be found here (it should be commentable by all):
> >
> https://docs.google.com/document/d/1Eps9eHvBc_qM8nRsTVwVCuWwHoEtQ-a-8Lv5dswuQoM/edit
> >
> > Any and all feedback is appreciated!
> >
> > A draft PR is up for Java/C++/Python, though it is far from complete:
> > https://github.com/apache/arrow/pull/4047
> >
> > [1]: https://github.com/apache/arrow/blob/master/format/Flight.proto
> > [2]: https://issues.apache.org/jira/browse/ARROW-4651
> >
> > Best,
> > David
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message