From user-return-151-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Mon Jul 8 07:57:12 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 22231180665 for ; Mon, 8 Jul 2019 09:57:12 +0200 (CEST) Received: (qmail 59353 invoked by uid 500); 8 Jul 2019 07:57:11 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 59337 invoked by uid 99); 8 Jul 2019 07:57:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Jul 2019 07:57:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id EFAC3C01EB for ; Mon, 8 Jul 2019 07:57:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.33 X-Spam-Level: *** X-Spam-Status: No, score=3.33 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, PDS_NO_HELO_DNS=1.327, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=353solutions-com.20150623.gappssmtp.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id RYcWzFWHUXaL for ; Mon, 8 Jul 2019 07:57:08 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::335; helo=mail-ot1-x335.google.com; envelope-from=miki@353solutions.com; receiver= Received: from mail-ot1-x335.google.com (mail-ot1-x335.google.com [IPv6:2607:f8b0:4864:20::335]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id E6ECD7E21D for ; Mon, 8 Jul 2019 07:57:07 +0000 (UTC) Received: by mail-ot1-x335.google.com with SMTP id q10so15229592otk.10 for ; Mon, 08 Jul 2019 00:57:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=353solutions-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=AE+RxqEUMBmuZHIvpsQXvya0KafOpVop31GpTp3lT/k=; b=JwwlPapHc+4bUG8DmrY76c66C+yteqphp3cCLkUERdYvdgBYIKJ0/ijeb4Axzxbh3W q9zvIxIV82hBopQpjXWQ/2yeE2nYKgPrHwX+meWnX+P6jpSX6b/C+0fmm8QLZR5qNAvr erijnANVE+J6/bsAVOO45hA8fjvcjcuIJoT12J6ijoWtM1AYdscmOLyi6q1Tj/wi8QS1 r/lRRDLF73EpUMWBXYlcMj/V5f1sLunTUuFZnuMlUkKb6ZHkUOS2Row6i/hKaiukLybq 1UOWsZUPEGsbefMNKXbVqKgHKFF2mLVbTF4WGwTHXfKk5Gz3gM31K/8bteUS4IuWs0nZ i9Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=AE+RxqEUMBmuZHIvpsQXvya0KafOpVop31GpTp3lT/k=; b=cBWueLXBW7qSp+PTJav+/jgCSOW5WS5o1pP13BoqYkAHXZrKGP+jVl53kpW92jJNeY oztWGu1D5T0nMZ+bBOHDnPu0Xuc5cwal/2dp8yGK1T5xG3ndb4Rt3VdVtPRWSrNwG4ku WF3FOFMApbw4WBvJ+P+m1286xG6utCQE2haTkUot4J9au4C818ub1TBiK5ARB+fvHAPW una6fIKqlmzlpkdH/0YyYrMBmbNiVP3EbzRuO/+2HPMXu9t9kLbDD4xHU+/XrNCm69Zq Q9UwZ52TTMhhVzpx7NZ0Qop3gkL4Q8sHdPBVVquyQs++V8V+nxgY8+hvui68/FnYGfsS xEyQ== X-Gm-Message-State: APjAAAXNEfAQ4r5MMuvNsGQZPxB0JPh5/uWVQlIFoljLNYtZJHNx2lc7 B+NLTRI7dQJzgGZ99rBr6y1MgPJy3RkQ4I+U82zIe+uL9Mk= X-Google-Smtp-Source: APXvYqwAsk567jflNy9Od02nLfnAKdzCwEMczAtgQJQehUKoXnfG0IqBTaJXWqj2iu5tusuBbqPcc6TF7G/wftgZpIs= X-Received: by 2002:a9d:23ca:: with SMTP id t68mr13249874otb.98.1562572620400; Mon, 08 Jul 2019 00:57:00 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Miki Tebeka Date: Mon, 8 Jul 2019 10:56:49 +0300 Message-ID: Subject: Re: Go / Python Sharing To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000f99c38058d26c925" --000000000000f99c38058d26c925 Content-Type: text/plain; charset="UTF-8" My bad, IPC in Go seems to be implemented - https://issues.apache.org/jira/browse/ARROW-3679 On Mon, Jul 8, 2019 at 10:18 AM Sebastien Binet wrote: > As far as i know, Go does support IPC (as in the arrow IPC format) > > Another option which has been discussed at some point was to have a shared > memory allocator so the arrow arrays could be shared between processes. > > I haven't looked in details what implementing plasma support for Go would > need on the Go side... > > -s > > > sent from my droid > > On Mon, Jul 8, 2019, 08:29 Miki Tebeka wrote: > >> Hi Clive, >> >> I'd like to understand the high level design for a system where a Go >>> process can communicate an Arrow data structure to a python process on the >>> same CPU >>> >> I see two options >> - Different processes with hared memory, probably using plasma >> - Same process. The either Go uses Python shared library or Python using >> Go compiled to shared library (-build-mode=c-shared) >> >> >>> - and for the python process to zero-copy gain access to that data, >>> change it and inform the Go process. This is low latency so I don't want >>> to save to file. >>> >> IIRC arrow is not built for mutation. You build an Array/Table once and >> then use it. >> >> Would this need the use of Plasma as a zero-copy store for the data >>> between the two processes or do I need to use IPC? But with IPC you are >>> transferring the data which is not needed in this case as I understand it. >>> Any pointers to examples would be appreciated. >>> >> See above about options. Note that currently the Go arrow implementation >> doesn't support IPC or plasma (though it's in the works). >> >> Yoni & I are working on another option which is using the C++ arrow >> library from Go. It does support plasma and since it uses the same >> underlying C++ library that Python does you'll be able to pass a pointer >> around without copying data. It's at very alpha-ish state but you're more >> than welcomed to give it a try - https://github.com/353solutions/carrow >> >> Happy hacking, >> Miki >> > --000000000000f99c38058d26c925 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
My bad, IPC in Go seems to be implemented -=C2=A0https://issues.ap= ache.org/jira/browse/ARROW-3679

On Mon, Jul 8, 2019 at 10:18 A= M Sebastien Binet <seb.binet@gmai= l.com> wrote:
As far as i know, Go does support IPC (as in the arr= ow IPC format)

Another option = which has been discussed at some point was to have a shared memory allocato= r so the arrow arrays could be shared between processes.

I haven't looked in details what imple= menting plasma support for Go would need on the Go side...

-s


sent from my droid

On Mon, Jul 8, 2019, 08:29 Miki T= ebeka <miki@3= 53solutions.com> wrote:
Hi Clive,

I'd like to understand the hig= h level design for a system where a Go process can communicate an Arrow dat= a structure to a python process on the same CPU
I see two options
- Different processes with hare= d memory, probably using plasma
- Same process. The either Go us= es Python shared library or Python using Go compiled to shared library (-bu= ild-mode=3Dc-shared)
=C2=A0
- and for the python process to zer= o-copy gain access to that data, change it and inform the Go process.=C2=A0= This is low latency so I don't want to save to file.
IIRC arrow is not built for mutation.=C2=A0You build an Array/Table once and then use it.

Would this need the use of Plasma as a zero-copy store for t= he data between the two processes or do I need to use IPC? But with IPC you= are transferring the data which is not needed in this case as I understand= it. Any pointers to examples would be appreciated.
See above about options. Note that currently the Go arrow impl= ementation doesn't support IPC or plasma (though it's in the works)= .

Yoni & I are wor= king on another option which is using the C++ arrow library from Go. It doe= s support plasma and since it uses the same underlying C++ library that Pyt= hon does you'll be able to pass a pointer around without copying data. = It's at very alpha-ish state but you're more than welcomed to give = it a try - https://github.com/353solutions/carrow

Happy hacking,
<= div>Miki=C2=A0
--000000000000f99c38058d26c925--