From user-return-7-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Sat Sep 1 20:47:13 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id E7DE9180630 for ; Sat, 1 Sep 2018 20:47:12 +0200 (CEST) Received: (qmail 95042 invoked by uid 500); 1 Sep 2018 18:47:12 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 95023 invoked by uid 99); 1 Sep 2018 18:47:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Sep 2018 18:47:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8CD51C00C7; Sat, 1 Sep 2018 18:47:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.131 X-Spam-Level: X-Spam-Status: No, score=-0.131 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id BQpktdmGUHRe; Sat, 1 Sep 2018 18:47:10 +0000 (UTC) Received: from mail-it0-f46.google.com (mail-it0-f46.google.com [209.85.214.46]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 144F35F343; Sat, 1 Sep 2018 18:47:10 +0000 (UTC) Received: by mail-it0-f46.google.com with SMTP id h1-v6so11100588itj.4; Sat, 01 Sep 2018 11:47:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=rPuKirau1qjhHvtBKzlFBiy+c7XWCbHEGpLu+nxsdqU=; b=sk96VkicS5huSIhO9NkjiBmFySkLn9dlrF+U9iyEvYBL2S+FnVY7UDuRaNVolsmBs3 u0mVD8yQGeK+1fAB13n8GEGgGQ0evn2kG38c0Eg3rYd4KZEiLDLVHreojzAY4ePoyLXu yYu34cKxxX5Ar0C9n9xgaSavO9o62oqtmSqo4vzNE3TjLF5vk+o18R7LwCLl486BGj5q lY5R8y9khy/7n8eEfYMQcA7hyoSoziaNO+4XFiNl9bvauluD8J8srVIxY3dP3PTZqqPT i766Zc1dFYi/XBlwlIPtArlYz3pUMc9urpncVnDBd5dfvxVz3mrn+5RkPYqamWTscL6R bTOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=rPuKirau1qjhHvtBKzlFBiy+c7XWCbHEGpLu+nxsdqU=; b=KD/bjEjxzS1stR01lOz0T41ax66tICGdsw3YS80ZwfkrInmxWjj/vba1kl6fVZHHNc 40gx7YZYqa2i/CvzNKecmBiyBJkdJEbPIbP+gbKfqaT9dmwJw7SPQon8LWJUg+RlLF/2 boa4Q2+DhxdDkARqayPxNlKviVl+KfQJNlRSFcU+e2UBQBhjOgcCAOQy3meYD5VH6Acs jf1GnZZUHeX539kPpj/VTbQN+0ZJSIDQJTvBtsPgTt3h1DZ27cUtfM1Bnomvqvp0OHn+ Fv/5PMv9znqGeaoHRogDLeu40JPtTmd/6CmjPFvFZV5sSKsmanxIfhF4Xkvaj0ra7TeC v2NQ== X-Gm-Message-State: APzg51AbCsGKNIbsgm9JL/mYGPUnxcvc9dx5xl445FEquUx4wQaoIgpm NOcCHRbQxEmiHSWSy/qV2xODFtiBWs6dv/gLDqYuwqlM X-Google-Smtp-Source: ANB0VdZNeVBe1dhOONQcJYMRcKYZeL13mSYvy/ylbm7Uzng0fZW2Aw9rS6Sn31t3u3SOyk5fjqnKp2dyxYWjChik9Us= X-Received: by 2002:a24:7b8d:: with SMTP id q135-v6mr1378225itc.43.1535827628413; Sat, 01 Sep 2018 11:47:08 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Wes McKinney Date: Sat, 1 Sep 2018 14:46:31 -0400 Message-ID: Subject: Re: IPC Example To: user@arrow.apache.org, dev@arrow.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable + dev@ There's several examples of sending record batches from Java to Python (and vice versa) over a network socket -- e.g. Jacques and I are working on a prototype of a general purpose Arrow-native RPC framework in Java and C++ respectively. Where there's some R&D needed is in Java interactions with shared memory. So if you want to do a zero copy read from a memory mapped file, then some development in the Arrow Java libraries is required. I'm not an expert but it seems like Netty has a mechanism to interact with ByteBuffer, which should include MappedByteBuffer https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffe= r/ReadOnlyUnsafeDirectByteBuf.java#L25 Correspondingly, an interface could be developed to enable the Java IPC code path to write to a shared memory region. To the Java developers, could we create some JIRA issues (if there are not already) around Java shared memory IPC? - Wes On Sat, Sep 1, 2018 at 2:32 PM ALBERTO Bocchinfuso wrote: > > I want to reinforce this request. I am interested in the same topic. > > I=E2=80=99d like an example specially focused on the creation of a Record= Batch passed from a Java program to a python one and vice-versa. > > > > Thanks, > > Alberto > > > > ________________________________ > Da: Clive Cox > Inviato: Saturday, September 1, 2018 6:12:29 PM > A: user@arrow.apache.org > Oggetto: IPC Example > > Hi, > > Is there any example of how to do say Java - Python IPC? I'm not sure ho= w to get started. > > I'm thinking of using Arrow IPC to replace REST/gRPC APIs for communicat= ion when everything can be run on a single computer node and low-latency is= the goal - hoping to remove the cost of serialization/deserialization and = network costs. Would this make sense. > > Thanks, > > Clive >