arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <...@cloudera.com>
Subject Re: Understanding "shared" memory implications
Date Tue, 15 Mar 2016 23:02:40 GMT
My understanding is that you can use java.nio.MappedByteBuffer to work
with memory-mapped files as one way to share memory pages between Java
(and non-Java) processes without copying.

I am hoping that we can reach a POC of zero-copy Arrow memory sharing
Java-to-Java and Java-to-C++ in the near future. Indeed this will have
huge implications once we get it working end to end (for example,
receiving memory from a Java process in Python without a heavy ser-de
step -- it's what we've always dreamed of) and with the metadata and
shared memory control flow standardized.

- Wes

On Wed, Mar 9, 2016 at 9:25 PM, Corey J Nolet <cjnolet@gmail.com> wrote:
> If I understand correctly, Arrow is using Netty underneath which is using Sun's Unsafe
API in order to allocate direct byte buffers off heap. It is using Netty to communicate between
"client" and "server", information about memory addresses for data that is being requested.
>
> I've never attempted to use the Unsafe API to access off heap memory that has been allocated
in one JVM from another JVM but I'm assuming this must be the case in order to claim that
the memory is being accessed directly without being copied, correct?
>
> The implication here is huge. If the memory is being directly shared across processes
by them being allowed to directly reach into the direct byte buffers, that's true shared memory.
Otherwise, if there's copies going on, it's less appealing.
>
>
> Thanks.
>
> Sent from my iPad

Mime
View raw message