From user-return-544-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Wed Jul 22 13:00:12 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 1CCDD180643 for ; Wed, 22 Jul 2020 15:00:12 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id 96AC6125FEC for ; Wed, 22 Jul 2020 12:59:42 +0000 (UTC) Received: (qmail 48007 invoked by uid 500); 22 Jul 2020 12:59:41 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 47997 invoked by uid 99); 22 Jul 2020 12:59:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2020 12:59:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id B98621A41BA for ; Wed, 22 Jul 2020 12:59:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.212 X-Spam-Level: X-Spam-Status: No, score=0.212 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=0.2, KAM_DMARC_STATUS=0.01, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id jUYqvz1-In6o for ; Wed, 22 Jul 2020 12:59:38 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=165.227.154.27; helo=zg8tmty1ljiyny4xntqumjca.icoremail.net; envelope-from=jamie@seu.edu.cn; receiver= Received: from zg8tmty1ljiyny4xntqumjca.icoremail.net (zg8tmty1ljiyny4xntqumjca.icoremail.net [165.227.154.27]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with SMTP id 6CC2E7F6D6 for ; Wed, 22 Jul 2020 12:59:36 +0000 (UTC) Received: from [IPv6:::ffff:192.168.0.2] (unknown [49.66.137.37]) by app2 (Coremail) with SMTP id cMhACgB3z758OBhfXTpEAQ--.33340S2; Wed, 22 Jul 2020 21:00:45 +0800 (CST) MIME-Version: 1.0 To: "user@arrow.apache.org" From: Xiaozhen Liu Subject: Does Arrow Flight use memory-mapped files for IPC within the same host? Date: Wed, 22 Jul 2020 20:59:23 +0800 Importance: normal X-Priority: 3 Content-Type: multipart/alternative; boundary="_22168D99-A93C-47AD-8538-2129043A0D77_" X-CM-TRANSID:cMhACgB3z758OBhfXTpEAQ--.33340S2 Message-Id:<5F18387D.00056B.24071@seu.edu.cn> X-Coremail-Antispam: 1UD129KBjvJXoW7AryfWryDtF4fXF17JF45Awb_yoW8WFyxpF WftwsxKrnrXF13Xr97Ar1xXF13WanYkrW5Krs7KrZxAwnxGFyS93y3Kw4Ykr1xGr4xW3yq qr1vyF98CayUZ3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUm0b7Iv0xC_KF4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Xr0_Ar1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4 A2jsIEc7CjxVAFwI0_GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG67k08I80 eVW5JVWrJwAqx4xG64kEw2xG04xIwI0_Gr0_Xr1l5I8CrVCF0I0E4I0vr24l5I8CrVC2j2 CEjI02ccxYII8I67AEr4CY67k08wAv7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAF wI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0Y48IcVAKI48JM4kE64xI4xA0e2IEY2 1l42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWU GVWUWwC20s026x8GjcxK67AKxVWUGVWUWwC2zVA0820Y0xCF62I06xkIj41lx4CE17CEb7 AF67AKxVWUJVWUXwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE 2Ix0cI8IcVCY1x0267AKxVWUJVW8JwCI42IY6xAIw20EY4v20xvaj40_WFyUJVCq3wCI42 IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r1j6r4UYxBIdaVFxhVj vjDU0xZFpf9x07jqQ6LUUUUU= X-CM-SenderInfo: ysrtilaqxsmqxvhxhvlgxou0/ --_22168D99-A93C-47AD-8538-2129043A0D77_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hi everyone, Lately, I=E2=80=99ve been experimenting with Arrow Flight. For now, I think= it is really great, especially when I=E2=80=99m not planning on building m= y own IPC framework (as I=E2=80=99ve mentioned earlier I=E2=80=99m trying t= o use Arrow to communicate between Java and Python processes). And the data= transfer speed is very satisfactory, although I haven=E2=80=99t tried very= big data. However, I=E2=80=99m wondering this: when I=E2=80=99m using Arrow Flight to= do IPC within the same machine, is there any kind of optimization? And by = optimization I mean will Flight internally use something like memory-mapped= files to transfer data? Because even though Flight optimizes speed, if it = still transfers data over the wire it cannot be faster than shared-memory (= file), right? I know this may be strange since Arrow Flight is an RPC framework and will = probably be better suited for communication between different hosts. But th= e fact that it also provides an RPC protocol that saves me the trouble of b= uilding my own IPC framework makes me choose Flight to do IPC (currently st= ill on the same host).=20 I know that KNIME Analytics Platform also uses Arrow for IPC, and it also u= ses temp Arrow file to transfer data. I can also do this within the framewo= rk of Arrow Flight by simply passing the location of temp files in the mess= ages. But first I just want to see if it is already implemented by Flight i= nternally.=20 I=E2=80=99ve looked up the source code of Flight and haven=E2=80=99t found = anything that looks like what I=E2=80=99m describing. Am I missing somethin= g, or is this the case, Flight doesn=E2=80=99t (and doesn=E2=80=99t plan to= ) use file for IPC within the same host? Thank you. Best, Xiaozhen Liu --_22168D99-A93C-47AD-8538-2129043A0D77_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8"

Hi everyone,

 

Lately, I=E2=80=99ve been experi= menting with Arrow Flight. For now, I think it is really great, especially = when I=E2=80=99m not planning on building my own IPC framework (as I=E2=80= =99ve mentioned earlier I=E2=80=99m trying to use Arrow to communicate betw= een Java and Python processes). And the data transfer speed is very satisfa= ctory, although I haven=E2=80=99t tried very big data.

However, I=E2=80=99m wondering this: when I=E2=80=99m using Arrow Fligh= t to do IPC within the same machine, is there any kind of optimization? And= by optimization I mean will Flight internally use something like memory-ma= pped files to transfer data? Because even though Flight optimizes speed, if= it still transfers data over the wire it cannot be fa= ster than shared-memory (file), right?

I kn= ow that KNIME Analytics Platform also uses Arrow for IPC, and it also uses = temp Arrow file to transfer data. I can also do this within the framework o= f Arrow Flight by simply passing the location of temp files in the messages= . But first I just want to see if it is already implemented by Flight inter= nally.

I=E2=80=99ve looked up the source code of Flight and haven=E2= =80=99t found anything that looks like what I=E2=80=99m describing. Am I mi= ssing something, or is this the case, Flight doesn=E2=80=99t (and doesn=E2= =80=99t plan to ) use file for IPC within the same host?

 <= /p>

Thank you.

 <= /p>

Best,

Xiaozhen Liu<= span style=3D'font-size:10.5pt;font-family:"Segoe UI",sans-serif;background= :white'>

 

= --_22168D99-A93C-47AD-8538-2129043A0D77_--