arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antoine Pitrou <anto...@python.org>
Subject Re: [Discuss][Format] Arrow Flight URI scheme proposal
Date Wed, 03 Apr 2019 08:36:43 GMT

I can.

Regards

Antoine.


Le 03/04/2019 à 02:32, Wes McKinney a écrit :
> I started a vote for the other Flight discussion thread, which will
> close on Friday. Since I'm about to leave on vacation can Antoine or
> Jacques run the vote for this one?
> 
> Thanks
> 
> On Tue, Apr 2, 2019 at 7:07 AM David Li <li.davidm96@gmail.com> wrote:
>>
>> Agreed with Antoine on grpc+tcp as the default. A gRPC server
>> generally won't offer both encrypted and unencrypted connections, so
>> this won't establish an insecure session where a secure one is
>> available. We could implement a TLS upgrade mechanism later as well.
>> I've updated the document to match.
>>
>> I'll also note that at least in preliminary Java benchmarks we
>> conducted, TLS had a very significant throughput cost in gRPC/Flight
>> (enough to wipe out any improvement Flight gave).
>>
>> Best,
>> David
>>
>> On 4/2/19, Antoine Pitrou <antoine@python.org> wrote:
>>>
>>> Le 02/04/2019 à 01:28, Jacques Nadeau a écrit :
>>>> My thinking is ideally the protocol would be more opaque than engineer-y
>>>> in
>>>> that an upgrade would happen as part of the negotiation process. For
>>>> example, when a connection is made, client says "hey, I also support
>>>> these
>>>> things" and then server responds and says "hey, let's send data on this
>>>> channel instead". That way, end consumers don't have to worry about the
>>>> details of alternative capabilities. (E.g. this is less about a Flight
>>>> endpoint).
>>>>
>>>> I like the protocol proposal with one small modification: I'd like us to
>>>> define a default protocol if one is not defined. I think it should
>>>> probably
>>>> be grpc+tls but am open to other options. What do others think about a
>>>> default protocol?
>>>
>>> That sounds reasonable, though IMHO it may be more useful if the default
>>> is the unencrypted version.
>>>
>>> (side note: while symmetric encryption is fast nowadays, it's not
>>> entirely costless)
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>>>
>>>>
>>>>
>>>> On Thu, Mar 28, 2019 at 9:29 PM Wes McKinney <wesmckinn@gmail.com>
wrote:
>>>>
>>>>> hi David,
>>>>>
>>>>> This seems like a reasonable evolution from where we are now. I will
>>>>> defer to others to comment on the low-level details
>>>>>
>>>>> This is sort of scope and kind of a can of worms, but one area where
>>>>> we should invest some thought is alternative FlightData transports,
>>>>> while allowing the "command layer" to continue to be gRPC. One such
>>>>> possible alternative scheme includes:
>>>>>
>>>>> * gRPC-over-TCP commands (actions, etc.)
>>>>> * Movement of IPC messages using RDMA (I have never actually used RDMA
>>>>> but it has been brought up to me as a topic of interest by multiple
>>>>> parties now)
>>>>>
>>>>> If a server supports an alternative protocols (e.g. gRPC-based for
>>>>> compatibility, plus RDMA for clients that implement it) then perhaps
>>>>> this information can be encoded in URIs. I'm not sure if there's prior
>>>>> art to look at on this design-wise
>>>>>
>>>>> - Wes
>>>>>
>>>>> On Wed, Mar 27, 2019 at 1:24 PM David Li <li.davidm96@gmail.com>
wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> We'd like to propose a URI scheme for Flight, in anticipation of
>>>>>> supporting multiple transports, and different configurations of the
>>>>>> gRPC transport. This will change Flight.proto[1] in format/ in a
>>>>>> backwards-incompatible way. This aims to fix ARROW-4651[2].
>>>>>>
>>>>>> The proposal can be found here (it should be commentable by all):
>>>>>>
>>>>> https://docs.google.com/document/d/1Eps9eHvBc_qM8nRsTVwVCuWwHoEtQ-a-8Lv5dswuQoM/edit
>>>>>>
>>>>>> Any and all feedback is appreciated!
>>>>>>
>>>>>> A draft PR is up for Java/C++/Python, though it is far from complete:
>>>>>> https://github.com/apache/arrow/pull/4047
>>>>>>
>>>>>> [1]: https://github.com/apache/arrow/blob/master/format/Flight.proto
>>>>>> [2]: https://issues.apache.org/jira/browse/ARROW-4651
>>>>>>
>>>>>> Best,
>>>>>> David
>>>>>
>>>>
>>>

Mime
View raw message