arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Tay <>
Subject Re: Understanding "shared" memory implications
Date Wed, 16 Mar 2016 05:50:50 GMT

I’m not sure whether past/current literature has a proper term for specific kind of IPC;
It would be very helpful to orientate present and future people on the codebase moving forward
But we don’t have to be pedantic right now … we can always revisit the issue when the
right time approaches…

Raymond Tay

-----Original Message-----
From: Jacques Nadeau <>
Reply-To: "" <>
Date: Wednesday, 16 March 2016 at 8:54 AM
To: "" <>
Subject: Re: Understanding "shared" memory implications

>The POC Steven and Wes are working on is based on MappedBuffer but I'm
>looking at using netty's fork of tcnative to use shared memory directly.
>We need to have both RPC and a shared memory mechanisms (what I'm inclined
>to call IPC but is a specific kind of IPC). The idea is we negotiate via
>RPC and then if we determine shared locality, we work over shared memory
>(preferably for both data and control). So the system interacting with
>HBase in your example would be the one responsible for placing collocated
>execution to take advantage of IPC.
>How do others feel of my redefinition of IPC to mean the same memory space
>communication (either via shared memory or rdma) versus RPC as socket based
>On Tue, Mar 15, 2016 at 5:38 PM, Corey Nolet <> wrote:
>> I was seeing Netty's unsafe classes being used here, not mapped byte
>> buffer  not sure if that statement is completely correct but I'll have to
>> dog through the code again to figure that out.
>> The more I was looking at unsafe, it makes sense why that would be
>> used.apparently it's also supposed to be included on Java 9 as a first
>> class API
>> On Mar 15, 2016 7:03 PM, "Wes McKinney" <> wrote:
>> > My understanding is that you can use java.nio.MappedByteBuffer to work
>> > with memory-mapped files as one way to share memory pages between Java
>> > (and non-Java) processes without copying.
>> >
>> > I am hoping that we can reach a POC of zero-copy Arrow memory sharing
>> > Java-to-Java and Java-to-C++ in the near future. Indeed this will have
>> > huge implications once we get it working end to end (for example,
>> > receiving memory from a Java process in Python without a heavy ser-de
>> > step -- it's what we've always dreamed of) and with the metadata and
>> > shared memory control flow standardized.
>> >
>> > - Wes
>> >
>> > On Wed, Mar 9, 2016 at 9:25 PM, Corey J Nolet <> wrote:
>> > > If I understand correctly, Arrow is using Netty underneath which is
>> > using Sun's Unsafe API in order to allocate direct byte buffers off heap.
>> > It is using Netty to communicate between "client" and "server",
>> information
>> > about memory addresses for data that is being requested.
>> > >
>> > > I've never attempted to use the Unsafe API to access off heap memory
>> > that has been allocated in one JVM from another JVM but I'm assuming this
>> > must be the case in order to claim that the memory is being accessed
>> > directly without being copied, correct?
>> > >
>> > > The implication here is huge. If the memory is being directly shared
>> > across processes by them being allowed to directly reach into the direct
>> > byte buffers, that's true shared memory. Otherwise, if there's copies
>> going
>> > on, it's less appealing.
>> > >
>> > >
>> > > Thanks.
>> > >
>> > > Sent from my iPad
>> >
View raw message