arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <wesmck...@gmail.com>
Subject Re: IPC Example
Date Sat, 01 Sep 2018 18:46:31 GMT
+ dev@

There's several examples of sending record batches from Java to Python
(and vice versa) over a network socket -- e.g. Jacques and I are
working on a prototype of a general purpose Arrow-native RPC framework
in Java and C++ respectively. Where there's some R&D needed is in Java
interactions with shared memory. So if you want to do a zero copy read
from a memory mapped file, then some development in the Arrow Java
libraries is required.

I'm not an expert but it seems like Netty has a mechanism to interact
with ByteBuffer, which should include MappedByteBuffer

https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/ReadOnlyUnsafeDirectByteBuf.java#L25

Correspondingly, an interface could be developed to enable the Java
IPC code path to write to a shared memory region.

To the Java developers, could we create some JIRA issues (if there are
not already) around Java shared memory IPC?

- Wes
On Sat, Sep 1, 2018 at 2:32 PM ALBERTO Bocchinfuso
<alberto_boc_94@hotmail.it> wrote:
>
> I want to reinforce this request. I am interested in the same topic.
>
> I’d like an example specially focused on the creation of a RecordBatch passed from
a Java program to a python one and vice-versa.
>
>
>
> Thanks,
>
> Alberto
>
>
>
> ________________________________
> Da: Clive Cox <cc@seldon.io>
> Inviato: Saturday, September 1, 2018 6:12:29 PM
> A: user@arrow.apache.org
> Oggetto: IPC Example
>
> Hi,
>
>  Is there any example of how to do say Java - Python IPC? I'm not sure how to get started.
>
>  I'm thinking of using Arrow IPC to replace REST/gRPC APIs for communication when everything
can be run on a single computer node and low-latency is the goal - hoping to remove the cost
of serialization/deserialization and network costs. Would this make sense.
>
>  Thanks,
>
>  Clive
>

Mime
View raw message