From user-return-8-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Sat Sep 1 21:15:18 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id A0DCB180630 for ; Sat, 1 Sep 2018 21:15:17 +0200 (CEST) Received: (qmail 18704 invoked by uid 500); 1 Sep 2018 19:15:16 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 18694 invoked by uid 99); 1 Sep 2018 19:15:16 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Sep 2018 19:15:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 5A9BBC00C7 for ; Sat, 1 Sep 2018 19:15:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.99 X-Spam-Level: * X-Spam-Status: No, score=1.99 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=quansight-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id HHWsTHgwAYMS for ; Sat, 1 Sep 2018 19:15:14 +0000 (UTC) Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 346345F117 for ; Sat, 1 Sep 2018 19:15:14 +0000 (UTC) Received: by mail-lj1-f170.google.com with SMTP id p6-v6so12523454ljc.5 for ; Sat, 01 Sep 2018 12:15:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quansight-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=LLqxNEqTw12yvJeTXSHgiQrm/wcGgluJWnDF/hxKdv0=; b=vc+3xOh8ITFBBS05JgPxvJXVGUJWFVkFHD6f3MuYVxXv324lr/rMoGSpupDVbBKVHl 9EeIlEllEs52x3mypOdzpNswSB9KernS6xuI6DSATb5PfjF2fkQPBgfvipwVgvK9KfFr 3P9tjd/NFucsIjZJmtu7UIAml/SMJwV5M1s2D7I5cYyGMGzDUTgp3E8Ae4zkCL2wqZqj 1FIDa6d/BYF0JJ2gM0q7W7OWZntkH8dVwSOpL3K4ZR5EhgO8cpOveZ5qdROOLxtXX6FE XRRIKEmoncGBS3cMZ8ZsC3jkew/VDkxAjDoh5aAAQtTPxxWba+prcMhXLu/4TYfCJUdL V8Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=LLqxNEqTw12yvJeTXSHgiQrm/wcGgluJWnDF/hxKdv0=; b=VLOvk1Ss1iRKljj3jOuHY9QTV8lbTwYnnyGPhuSmdtS5fn9c65vqmkbz6cQPY6obr5 XMfKO9wW+npqD4Zg3jDTUoXdVRYU6rEVZQbYavd9BKP+i1DGpuNwalbOTFzvYaglwEz+ zImS4uUBzWtFNfvgkpj7S1EJp1Rz0X/kWFfTROEpSsamAWMmEBHOA6xmGyi8ZPk3uzUA XOuMITcf8mqH8/DuxlaOLzDJU7nVsb5THaXh435Ah/5LcXm0mWchvjgrmmZRMlwdByhr twcd9Zz3lPTDNVh2C1pn7EPk4xqfE4D4gajWjENldekJmHEKFssynToz4O3lmjmyvZwR P57w== X-Gm-Message-State: APzg51AVXsZH5OBT1ZROCKpgpmsEkkHpFLnW0YqxphE/NH3ucVNFSyWR H+wgDXQpmyGNJ3CDN1alDxBd0noeiGbxKLOllLWgkLb3eXc= X-Google-Smtp-Source: ANB0VdbkRZL8iDsghriFYm4ERbiusGegR+aM9UXWhLNoaOXO0Sk/7pafOSP1L/Tmwn07XSIXt9JOJTPP7pczCophJAw= X-Received: by 2002:a2e:800e:: with SMTP id j14-v6mr13283350ljg.114.1535829313674; Sat, 01 Sep 2018 12:15:13 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a2e:8b:0:0:0:0:0 with HTTP; Sat, 1 Sep 2018 12:15:13 -0700 (PDT) In-Reply-To: References: From: Pearu Peterson Date: Sat, 1 Sep 2018 22:15:13 +0300 Message-ID: Subject: Re: IPC Example To: dev@arrow.apache.org, user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000ad75e50574d42013" --000000000000ad75e50574d42013 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, I'd also like to enforce the raised question, in particular, it would be very useful to have basic examples of IPC between same or different languages, including C/C++, Python, Java, etc. Whatever combination of languages is used, the principles of IPC should be the same. For instance, in Python-Java or Python-Python IPC cases, the Python code should not depend on in what language is written the code running another process. Is this understanding correct? Btw, while reading the arrow tests, I noticed a comment IPC only supported on Linux in https://github.com/apache/arrow/blob/master/cpp/src/arrow/gpu/cuda-test.cc#= L126 Does this restriction apply only for CUDA IPC or is the comment more general? What would it take to add IPC support for Windows or OSX? Best regards, Pearu On Sat, Sep 1, 2018 at 9:46 PM, Wes McKinney wrote: > + dev@ > > There's several examples of sending record batches from Java to Python > (and vice versa) over a network socket -- e.g. Jacques and I are > working on a prototype of a general purpose Arrow-native RPC framework > in Java and C++ respectively. Where there's some R&D needed is in Java > interactions with shared memory. So if you want to do a zero copy read > from a memory mapped file, then some development in the Arrow Java > libraries is required. > > I'm not an expert but it seems like Netty has a mechanism to interact > with ByteBuffer, which should include MappedByteBuffer > > https://github.com/netty/netty/blob/4.1/buffer/src/ > main/java/io/netty/buffer/ReadOnlyUnsafeDirectByteBuf.java#L25 > > Correspondingly, an interface could be developed to enable the Java > IPC code path to write to a shared memory region. > > To the Java developers, could we create some JIRA issues (if there are > not already) around Java shared memory IPC? > > - Wes > On Sat, Sep 1, 2018 at 2:32 PM ALBERTO Bocchinfuso > wrote: > > > > I want to reinforce this request. I am interested in the same topic. > > > > I=E2=80=99d like an example specially focused on the creation of a Reco= rdBatch > passed from a Java program to a python one and vice-versa. > > > > > > > > Thanks, > > > > Alberto > > > > > > > > ________________________________ > > Da: Clive Cox > > Inviato: Saturday, September 1, 2018 6:12:29 PM > > A: user@arrow.apache.org > > Oggetto: IPC Example > > > > Hi, > > > > Is there any example of how to do say Java - Python IPC? I'm not sure > how to get started. > > > > I'm thinking of using Arrow IPC to replace REST/gRPC APIs for > communication when everything can be run on a single computer node and > low-latency is the goal - hoping to remove the cost of > serialization/deserialization and network costs. Would this make sense. > > > > Thanks, > > > > Clive > > > --000000000000ad75e50574d42013 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I'd also= like to enforce=C2=A0the=C2=A0raised question, in particular, it would be = very useful to have basic examples of IPC between same or different languag= es, including C/C++, Python, Java, etc.=C2=A0

What= ever combination of languages is used, the principles of IPC should be the = same. For instance, in Python-Java or Python-Python IPC cases, the Python c= ode should not depend on in what language is written the code running anoth= er process. Is this understanding correct?

Btw, while rea= ding the arrow tests, I noticed a comment

=C2=A0 IPC only= supported on Linux

in https://github.com/ap= ache/arrow/blob/master/cpp/src/arrow/gpu/cuda-test.cc#L126

Does = this restriction apply only for CUDA IPC or is the comment more general?What would it take to add IPC support for Windows or OSX?

Best rega= rds,
Pearu

On Sat, Sep 1, 2018 at 9:46 PM, Wes McKinney <wesmc= kinn@gmail.com> wrote:
+ de= v@

There's several examples of sending record batches from Java to Python<= br> (and vice versa) over a network socket -- e.g. Jacques and I are
working on a prototype of a general purpose Arrow-native RPC framework
in Java and C++ respectively. Where there's some R&D needed is in J= ava
interactions with shared memory. So if you want to do a zero copy read
from a memory mapped file, then some development in the Arrow Java
libraries is required.

I'm not an expert but it seems like Netty has a mechanism to interact with ByteBuffer, which should include MappedByteBuffer

https://github.com/netty/netty/blob/4.1/buffer/src/m= ain/java/io/netty/buffer/ReadOnlyUnsafeDirectByteBuf.java#L25=

Correspondingly, an interface could be developed to enable the Java
IPC code path to write to a shared memory region.

To the Java developers, could we create some JIRA issues (if there are
not already) around Java shared memory IPC?

- Wes
On Sat, Sep 1, 2018 a= t 2:32 PM ALBERTO Bocchinfuso
<alberto_boc_94@hotmail.it<= /a>> wrote:
>
> I want to reinforce this request. I am interested in the same topic. >
> I=E2=80=99d like an example specially focused on the creation of a Rec= ordBatch passed from a Java program to a python one and vice-versa.
>
>
>
> Thanks,
>
> Alberto
>
>
>
> ________________________________
> Da: Clive Cox <cc@seldon.io>=
> Inviato: Saturday, September 1, 2018 6:12:29 PM
> A: user@arrow.apache.org<= br> > Oggetto: IPC Example
>
> Hi,
>
>=C2=A0 Is there any example of how to do say Java - Python IPC? I'm= not sure how to get started.
>
>=C2=A0 I'm thinking of using Arrow IPC to replace REST/gRPC APIs fo= r communication when everything can be run on a single computer node and lo= w-latency is the goal - hoping to remove the cost of serialization/deserial= ization and network costs. Would this make sense.
>
>=C2=A0 Thanks,
>
>=C2=A0 Clive
>

--000000000000ad75e50574d42013--