arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Li <li.david...@gmail.com>
Subject Re: [Discuss][Format] Arrow Flight URI scheme proposal
Date Tue, 02 Apr 2019 12:07:46 GMT
Agreed with Antoine on grpc+tcp as the default. A gRPC server
generally won't offer both encrypted and unencrypted connections, so
this won't establish an insecure session where a secure one is
available. We could implement a TLS upgrade mechanism later as well.
I've updated the document to match.

I'll also note that at least in preliminary Java benchmarks we
conducted, TLS had a very significant throughput cost in gRPC/Flight
(enough to wipe out any improvement Flight gave).

Best,
David

On 4/2/19, Antoine Pitrou <antoine@python.org> wrote:
>
> Le 02/04/2019 à 01:28, Jacques Nadeau a écrit :
>> My thinking is ideally the protocol would be more opaque than engineer-y
>> in
>> that an upgrade would happen as part of the negotiation process. For
>> example, when a connection is made, client says "hey, I also support
>> these
>> things" and then server responds and says "hey, let's send data on this
>> channel instead". That way, end consumers don't have to worry about the
>> details of alternative capabilities. (E.g. this is less about a Flight
>> endpoint).
>>
>> I like the protocol proposal with one small modification: I'd like us to
>> define a default protocol if one is not defined. I think it should
>> probably
>> be grpc+tls but am open to other options. What do others think about a
>> default protocol?
>
> That sounds reasonable, though IMHO it may be more useful if the default
> is the unencrypted version.
>
> (side note: while symmetric encryption is fast nowadays, it's not
> entirely costless)
>
> Regards
>
> Antoine.
>
>
>>
>>
>>
>> On Thu, Mar 28, 2019 at 9:29 PM Wes McKinney <wesmckinn@gmail.com> wrote:
>>
>>> hi David,
>>>
>>> This seems like a reasonable evolution from where we are now. I will
>>> defer to others to comment on the low-level details
>>>
>>> This is sort of scope and kind of a can of worms, but one area where
>>> we should invest some thought is alternative FlightData transports,
>>> while allowing the "command layer" to continue to be gRPC. One such
>>> possible alternative scheme includes:
>>>
>>> * gRPC-over-TCP commands (actions, etc.)
>>> * Movement of IPC messages using RDMA (I have never actually used RDMA
>>> but it has been brought up to me as a topic of interest by multiple
>>> parties now)
>>>
>>> If a server supports an alternative protocols (e.g. gRPC-based for
>>> compatibility, plus RDMA for clients that implement it) then perhaps
>>> this information can be encoded in URIs. I'm not sure if there's prior
>>> art to look at on this design-wise
>>>
>>> - Wes
>>>
>>> On Wed, Mar 27, 2019 at 1:24 PM David Li <li.davidm96@gmail.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> We'd like to propose a URI scheme for Flight, in anticipation of
>>>> supporting multiple transports, and different configurations of the
>>>> gRPC transport. This will change Flight.proto[1] in format/ in a
>>>> backwards-incompatible way. This aims to fix ARROW-4651[2].
>>>>
>>>> The proposal can be found here (it should be commentable by all):
>>>>
>>> https://docs.google.com/document/d/1Eps9eHvBc_qM8nRsTVwVCuWwHoEtQ-a-8Lv5dswuQoM/edit
>>>>
>>>> Any and all feedback is appreciated!
>>>>
>>>> A draft PR is up for Java/C++/Python, though it is far from complete:
>>>> https://github.com/apache/arrow/pull/4047
>>>>
>>>> [1]: https://github.com/apache/arrow/blob/master/format/Flight.proto
>>>> [2]: https://issues.apache.org/jira/browse/ARROW-4651
>>>>
>>>> Best,
>>>> David
>>>
>>
>

Mime
View raw message