From user-return-9-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Sat Sep 1 21:21:16 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4BE86180630 for ; Sat, 1 Sep 2018 21:21:16 +0200 (CEST) Received: (qmail 25928 invoked by uid 500); 1 Sep 2018 19:21:15 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 25909 invoked by uid 99); 1 Sep 2018 19:21:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Sep 2018 19:21:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0DB3E1805BD; Sat, 1 Sep 2018 19:21:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.111 X-Spam-Level: X-Spam-Status: No, score=-0.111 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id vLNVqO5LyR8Y; Sat, 1 Sep 2018 19:21:13 +0000 (UTC) Received: from mail-it0-f50.google.com (mail-it0-f50.google.com [209.85.214.50]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 068F95F117; Sat, 1 Sep 2018 19:21:13 +0000 (UTC) Received: by mail-it0-f50.google.com with SMTP id j81-v6so11171962ite.0; Sat, 01 Sep 2018 12:21:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=qav8lV8cgd+Z4uUD0aQtDCCMWcKv78AO6UILxD0Fung=; b=fyUnn3X02ikZ7txCiu5A0NPLFK1gL36YsUDMVenMGNVjnpTKPkBH/An9rQw2RROkt9 4VYmSPlh/gzJgLZm1vVNmWuW5q1xnSioK1cqCNurnECbj9s7O0NLGBZs+AaooFWRord+ Tfp07Ll1drjR9c6YuHL8nTbuCYdXLsLYr74lX7CH9wDvnhCOuR08IqdjFtQj7MY3pOwz go+wn9T5frGz6ZdTWZl+TEJznx8ycBjOJFW2mgWKli2ntLKwrfYWM7VJ+wIRGq0L4GvX eyVAedZSfTcjCiYxn0PN6zFUM6H9vjOfm98Gbsz18/mp4aNaAW7SytOXEssxxfYqYLFp fDgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=qav8lV8cgd+Z4uUD0aQtDCCMWcKv78AO6UILxD0Fung=; b=caf6sVd53F8bmv2kw2tooDBWO+VU4ZzZQZ0DPEYG2iukVLrvn8/10BEKsXqCWFr50b Iv6dT1AFAlbbuO1ziG2GEwzbHy1kH0ORICzyl5HffjNBCjXurxOYmZOq0M/NBsCkm9aw WZsejGFtQvOg0otS36+8FXZCO0xPZqMDbpTtuegHd7C3bDtYv5u/dTN9lf8l9gJoe+vl PtSmbzts/rvJlX5AlrpZBwl+kDgNgsk0fXaowS8eH9Ui7jf1sxTQXu84DaYylG0kvnrP wpucYx51ce9WcHtNwnxAsoIKa6JYwV3HD6pQu6kwoAgEdz/d9ncIAyHn5Uh2wmJZ0j4/ PO/w== X-Gm-Message-State: APzg51AO3UApZRv7VS+9btW5UI08G/6370jbybJkoCoJUiEX7uWFfgFu RYWO/fDAyK/+kzRup2wCTCAPzE/l7bJ3LPROCCre4L9W X-Google-Smtp-Source: ANB0VdbAM5Orajqy0dYaw8ZbAI+BxyT1FpW6sWlAa0bfTXhPa5qnc/s6rK5m+SMdys+EvQULT4ESSAWMQNH+/3WaQ8w= X-Received: by 2002:a24:4207:: with SMTP id i7-v6mr1425425itb.24.1535829665540; Sat, 01 Sep 2018 12:21:05 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Wes McKinney Date: Sat, 1 Sep 2018 15:20:29 -0400 Message-ID: Subject: Re: IPC Example To: user@arrow.apache.org Cc: dev@arrow.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable hi Pearu, On Sat, Sep 1, 2018 at 3:15 PM Pearu Peterson wrote: > > Hi, > > I'd also like to enforce the raised question, in particular, it would be = very useful to have basic examples of IPC between same or different languag= es, including C/C++, Python, Java, etc. > > Whatever combination of languages is used, the principles of IPC should b= e the same. For instance, in Python-Java or Python-Python IPC cases, the Py= thon code should not depend on in what language is written the code running= another process. Is this understanding correct? Right, the IPC protocol is not language dependent; this is one of the raison d'=C3=AAtres of this project. > > Btw, while reading the arrow tests, I noticed a comment > > IPC only supported on Linux > > in https://github.com/apache/arrow/blob/master/cpp/src/arrow/gpu/cuda-tes= t.cc#L126 > > Does this restriction apply only for CUDA IPC or is the comment more gene= ral? > What would it take to add IPC support for Windows or OSX? This only applies to CUDA IPC. I tried to find a definitive reference but straight from NVIDIA: https://github.com/NVIDIA/cuda-samples#cuda-interprocess-communication We test shared memory IPC on all three platforms in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-read-writ= e-test.cc - Wes > > Best regards, > Pearu > > On Sat, Sep 1, 2018 at 9:46 PM, Wes McKinney wrote: >> >> + dev@ >> >> There's several examples of sending record batches from Java to Python >> (and vice versa) over a network socket -- e.g. Jacques and I are >> working on a prototype of a general purpose Arrow-native RPC framework >> in Java and C++ respectively. Where there's some R&D needed is in Java >> interactions with shared memory. So if you want to do a zero copy read >> from a memory mapped file, then some development in the Arrow Java >> libraries is required. >> >> I'm not an expert but it seems like Netty has a mechanism to interact >> with ByteBuffer, which should include MappedByteBuffer >> >> https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/bu= ffer/ReadOnlyUnsafeDirectByteBuf.java#L25 >> >> Correspondingly, an interface could be developed to enable the Java >> IPC code path to write to a shared memory region. >> >> To the Java developers, could we create some JIRA issues (if there are >> not already) around Java shared memory IPC? >> >> - Wes >> On Sat, Sep 1, 2018 at 2:32 PM ALBERTO Bocchinfuso >> wrote: >> > >> > I want to reinforce this request. I am interested in the same topic. >> > >> > I=E2=80=99d like an example specially focused on the creation of a Rec= ordBatch passed from a Java program to a python one and vice-versa. >> > >> > >> > >> > Thanks, >> > >> > Alberto >> > >> > >> > >> > ________________________________ >> > Da: Clive Cox >> > Inviato: Saturday, September 1, 2018 6:12:29 PM >> > A: user@arrow.apache.org >> > Oggetto: IPC Example >> > >> > Hi, >> > >> > Is there any example of how to do say Java - Python IPC? I'm not sure= how to get started. >> > >> > I'm thinking of using Arrow IPC to replace REST/gRPC APIs for communi= cation when everything can be run on a single computer node and low-latency= is the goal - hoping to remove the cost of serialization/deserialization a= nd network costs. Would this make sense. >> > >> > Thanks, >> > >> > Clive >> > > >