arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miki Tebeka <m...@353solutions.com>
Subject Re: Go / Python Sharing
Date Mon, 08 Jul 2019 06:28:54 GMT
Hi Clive,

I'd like to understand the high level design for a system where a Go
> process can communicate an Arrow data structure to a python process on the
> same CPU
>
I see two options
- Different processes with hared memory, probably using plasma
- Same process. The either Go uses Python shared library or Python using Go
compiled to shared library (-build-mode=c-shared)


> - and for the python process to zero-copy gain access to that data, change
> it and inform the Go process.  This is low latency so I don't want to save
> to file.
>
IIRC arrow is not built for mutation. You build an Array/Table once and
then use it.

Would this need the use of Plasma as a zero-copy store for the data between
> the two processes or do I need to use IPC? But with IPC you are
> transferring the data which is not needed in this case as I understand it.
> Any pointers to examples would be appreciated.
>
See above about options. Note that currently the Go arrow implementation
doesn't support IPC or plasma (though it's in the works).

Yoni & I are working on another option which is using the C++ arrow library
from Go. It does support plasma and since it uses the same underlying C++
library that Python does you'll be able to pass a pointer around without
copying data. It's at very alpha-ish state but you're more than welcomed to
give it a try - https://github.com/353solutions/carrow

Happy hacking,
Miki

Mime
View raw message