Mailing-List: contact dev-help@arrow.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@arrow.apache.org
MIME-Version: 1.0
In-Reply-To: <72A2BCFD-54D7-4376-8199-04A5535D86C0@gmail.com>
References: <72A2BCFD-54D7-4376-8199-04A5535D86C0@gmail.com>
From: Wes McKinney <wes@cloudera.com>
Date: Tue, 15 Mar 2016 16:02:40 -0700
Message-ID: 
 <CADHNq9-MzgPPpLo1s5Of7uRE6dyWtws7+d1rMLOear0_evN_Ow@mail.gmail.com>
Subject: Re: Understanding "shared" memory implications
To: dev@arrow.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

My understanding is that you can use java.nio.MappedByteBuffer to work
with memory-mapped files as one way to share memory pages between Java
(and non-Java) processes without copying.

I am hoping that we can reach a POC of zero-copy Arrow memory sharing
Java-to-Java and Java-to-C++ in the near future. Indeed this will have
huge implications once we get it working end to end (for example,
receiving memory from a Java process in Python without a heavy ser-de
step -- it's what we've always dreamed of) and with the metadata and
shared memory control flow standardized.

- Wes

On Wed, Mar 9, 2016 at 9:25 PM, Corey J Nolet <cjnolet@gmail.com> wrote:
> If I understand correctly, Arrow is using Netty underneath which is using=
 Sun's Unsafe API in order to allocate direct byte buffers off heap. It is =
using Netty to communicate between "client" and "server", information about=
 memory addresses for data that is being requested.
>
> I've never attempted to use the Unsafe API to access off heap memory that=
 has been allocated in one JVM from another JVM but I'm assuming this must =
be the case in order to claim that the memory is being accessed directly wi=
thout being copied, correct?
>
> The implication here is huge. If the memory is being directly shared acro=
ss processes by them being allowed to directly reach into the direct byte b=
uffers, that's true shared memory. Otherwise, if there's copies going on, i=
t's less appealing.
>
>
> Thanks.
>
> Sent from my iPad