arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miki Tebeka <m...@353solutions.com>
Subject Re: Go / Python Sharing
Date Mon, 08 Jul 2019 07:56:49 GMT
My bad, IPC in Go seems to be implemented -
https://issues.apache.org/jira/browse/ARROW-3679

On Mon, Jul 8, 2019 at 10:18 AM Sebastien Binet <seb.binet@gmail.com> wrote:

> As far as i know, Go does support IPC (as in the arrow IPC format)
>
> Another option which has been discussed at some point was to have a shared
> memory allocator so the arrow arrays could be shared between processes.
>
> I haven't looked in details what implementing plasma support for Go would
> need on the Go side...
>
> -s
>
>
> sent from my droid
>
> On Mon, Jul 8, 2019, 08:29 Miki Tebeka <miki@353solutions.com> wrote:
>
>> Hi Clive,
>>
>> I'd like to understand the high level design for a system where a Go
>>> process can communicate an Arrow data structure to a python process on the
>>> same CPU
>>>
>> I see two options
>> - Different processes with hared memory, probably using plasma
>> - Same process. The either Go uses Python shared library or Python using
>> Go compiled to shared library (-build-mode=c-shared)
>>
>>
>>> - and for the python process to zero-copy gain access to that data,
>>> change it and inform the Go process.  This is low latency so I don't want
>>> to save to file.
>>>
>> IIRC arrow is not built for mutation. You build an Array/Table once and
>> then use it.
>>
>> Would this need the use of Plasma as a zero-copy store for the data
>>> between the two processes or do I need to use IPC? But with IPC you are
>>> transferring the data which is not needed in this case as I understand it.
>>> Any pointers to examples would be appreciated.
>>>
>> See above about options. Note that currently the Go arrow implementation
>> doesn't support IPC or plasma (though it's in the works).
>>
>> Yoni & I are working on another option which is using the C++ arrow
>> library from Go. It does support plasma and since it uses the same
>> underlying C++ library that Python does you'll be able to pass a pointer
>> around without copying data. It's at very alpha-ish state but you're more
>> than welcomed to give it a try - https://github.com/353solutions/carrow
>>
>> Happy hacking,
>> Miki
>>
>

Mime
View raw message