arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <>
Subject Re: IPC Example
Date Sat, 01 Sep 2018 18:46:31 GMT
+ dev@

There's several examples of sending record batches from Java to Python
(and vice versa) over a network socket -- e.g. Jacques and I are
working on a prototype of a general purpose Arrow-native RPC framework
in Java and C++ respectively. Where there's some R&D needed is in Java
interactions with shared memory. So if you want to do a zero copy read
from a memory mapped file, then some development in the Arrow Java
libraries is required.

I'm not an expert but it seems like Netty has a mechanism to interact
with ByteBuffer, which should include MappedByteBuffer

Correspondingly, an interface could be developed to enable the Java
IPC code path to write to a shared memory region.

To the Java developers, could we create some JIRA issues (if there are
not already) around Java shared memory IPC?

- Wes
On Sat, Sep 1, 2018 at 2:32 PM ALBERTO Bocchinfuso
<> wrote:
> I want to reinforce this request. I am interested in the same topic.
> I’d like an example specially focused on the creation of a RecordBatch passed from
a Java program to a python one and vice-versa.
> Thanks,
> Alberto
> ________________________________
> Da: Clive Cox <>
> Inviato: Saturday, September 1, 2018 6:12:29 PM
> A:
> Oggetto: IPC Example
> Hi,
>  Is there any example of how to do say Java - Python IPC? I'm not sure how to get started.
>  I'm thinking of using Arrow IPC to replace REST/gRPC APIs for communication when everything
can be run on a single computer node and low-latency is the goal - hoping to remove the cost
of serialization/deserialization and network costs. Would this make sense.
>  Thanks,
>  Clive

View raw message