Hello all,
<= /div>

I've been using the in-process sharing method f= or quite some time for the Python<->Java interaction and I really = like the ease of doing it all in the same process. Especially as this av= oids any memory-copy or shared memory handling. This is really useful fo= r the case where you only want to call a single routine in another langu= age.

Thus I would really like to see this a= lso implemented for Go (and Rust) so that one can build custom UDFs in i= t and use them from Python code. The pre-conditions for this are that we= have IPC tests that verify that both libraries use the exact same memor= y layout and that we can pull out the memory pointer from the Go Arrow s= tructures into the C++ memory structures and also keep a reference betwe= en both so that memory tracking doesn't deallocate the underlying memory= . For that we have in Python the pyarrow.foreign_buffer https://github.com/apache/arrow= /blob/1b798a317df719d32312ca2c3253a2e399e949b8/python/pyarrow/io.pxi#L12= 76-L1292 function.

For the Go<-= >Python case, I would though recommend to solve this as a Go<->= C++ interface as this would make interaction for all the libraries based= on the C++ one (like R, Ruby, ..) possible.

Uw= e

On Mon, Jul 8, 2019, at 9:57 AM, Miki Tebeka = wrote:

My bad, IPC in Go seems to be implemented - https://issues.apache.o= rg/jira/browse/ARROW-3679

On Mon, Jul 8,= 2019 at 10:18 AM Sebastien Binet <seb.binet@gmail.com> wrote:

= As far as i know, Go does support IPC (as in the arrow IPC format)

Another option which h= as been discussed at some point was to have a shared memory allocator so= the arrow arrays could be shared between processes.

I haven't looked in details what impl= ementing plasma support for Go would need on the Go side...

-s

sent from my droid
<= /div>

On Mon, Jul 8, 2019, 08:29 Miki Tebeka &l= t;miki@353solutions.com>= wrote:
Hi Cliv= e,

=
=
I'd like to understand the high level design for a= system where a Go process can communicate an Arrow data structure to a = python process on the same CPU
I see two options
- Different process= es with hared memory, probably using plasma
- Sam= e process. The either Go uses Python shared library or Python using Go c= ompiled to shared library (-build-mode=3Dc-shared)
<= br>
- and for the python process to zero-copy= gain access to that data, change it and inform the Go process. Th= is is low latency so I don't want to save to file.
IIRC arrow is not built for= mutation. You build= an Array/Table once and then use it.

Would this need the use of Plasma as a zero-copy store for the data bet= ween the two processes or do I need to use IPC? But with IPC you are tra= nsferring the data which is not needed in this case as I understand it. = Any pointers to examples would be appreciated.
See above about options. Note t= hat currently the Go arrow implementation doesn't support IPC or plasma = (though it's in the works).

Yoni & I are working on another option which is using the C++= arrow library from Go. It does support plasma and since it uses the sam= e underlying C++ library that Python does you'll be able to pass a point= er around without copying data. It's at very alpha-ish state but you're = more than welcomed to give it a try - https://github.com/353solutions/carr= ow

Happy hacking,=
Miki