arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <wesmck...@gmail.com>
Subject Re: Arrow for low latency IPC
Date Fri, 01 Nov 2019 15:47:18 GMT
hi Sam -- could you clarify a bit more about your use case?

Flight is used for passing batches efficiently over TCP (via gRPC). We
have shared memory IPC utilities as well, but coordination and
signaling / semaphores between processes are up to you. Plasma might
be worth exploring as a tool to help

- Wes

On Fri, Nov 1, 2019 at 3:32 AM Fan Liya <liya.fan03@gmail.com> wrote:
>
> Hi Samrat,
>
> Arrow has flexible support for IPC through grpc.
>
> The cpp benchmark can be found in:
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc
>
> The java benchmark can be found in:
> https://github.com/apache/arrow/blob/master/java/flight/src/test/java/org/apache/arrow/flight/perf/TestPerf.java
>
> About the latency problem, I am not clear about your scenario. Can you
> reduce the latency by making the batch small enough?
>
> Best,
> Liya Fan
>
> On Fri, Nov 1, 2019 at 10:58 AM Samrat Batth <samratbatth@gmail.com> wrote:
>
> > Hi,
> >
> > I am new to Apache Arrow and want to consider it for low latency IPC
> > stream. I want to see if I can use Arrow for low latency IPC via processes
> > written in multiple languages.
> >
> > As far as what I can understand about current version of Arrow, the
> > messages are passed in a batch via IPC, ie the RX will get the msgs in
> > batch and not as soon as they are sent by the TX.
> >
> > Any pointers to examples / code for this? Or any benchmarks on IPC?
> >
> > Thanks,
> > Sam

Mime
View raw message