From user-return-537-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Tue Jul 21 09:57:39 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 9944618065B for ; Tue, 21 Jul 2020 11:57:39 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id 529FB12519F for ; Tue, 21 Jul 2020 09:57:38 +0000 (UTC) Received: (qmail 60104 invoked by uid 500); 21 Jul 2020 09:57:37 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 60093 invoked by uid 99); 21 Jul 2020 09:57:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2020 09:57:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 1FCE3181418 for ; Tue, 21 Jul 2020 09:57:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.01 X-Spam-Level: X-Spam-Status: No, score=0.01 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id dKqM8rLxG5hg for ; Tue, 21 Jul 2020 09:57:36 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.166.42; helo=mail-io1-f42.google.com; envelope-from=hello.wjx@gmail.com; receiver= Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id CAF0FBB946 for ; Tue, 21 Jul 2020 09:57:35 +0000 (UTC) Received: by mail-io1-f42.google.com with SMTP id v8so20730956iox.2 for ; Tue, 21 Jul 2020 02:57:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=FU8+1zcDthbsxShoBKTMuiGPj7C5SshDp3HuWI6T/Dc=; b=atrLxf33to+nk0W5TwcRfKJWXK14V26tPANIPGNnVXAY+PMsR/ZJlkh3c/vZ7UZ8R/ ULMsvCv5eF4m9lD6K/BitZhLtBmCPFor8ErcVzKPyAhjlR3bo6cWscYMEPO3lBO9sXaC 3pQHnRmbed0EXuAksf0QcMQD3T2zoqQQ0candlgEmjfoD3CC2wvs+lnh9VwmlXJX4bbz AN68DOIU4GS4APIUOSOKtmE3XkHpmJaTVqkIrwAd3Ze28u71JZJ9kN0m8VcFxE153OsZ 6JvbAUbtAJlgNPxYqZeMqZwCXqLM3iJ7IkV3xcf/8bIhF44/i5sHZOfebdufAEV9aDhR YBbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=FU8+1zcDthbsxShoBKTMuiGPj7C5SshDp3HuWI6T/Dc=; b=SVo4AUZSe8WVUndJLVivw5vlSVrYqH0CVhV2Httpsvv1Yg8srDyW968JJR0mGZ6SiV yjaKTh6pzxLovH8kfhoG+gSE4MYDj+/8Gci/t+c0uq1lG/9tShE+1tRXQ7nLvaeEoMxS eCREm/43cB03patKZzqI/slN+WFXkeJ/FIx3N9JfoMM07a6bT0caDPwHBDdKNSBS2lAs ZOQOal4AOGYCnc7Arw61siJKByJ6k1XYqWSG/cT4gLMHixprTqbUvPXvbSo/mhen5SNn n9wCSXGt5YTXyluRlLs5w+rhfj6xzjFSajqshRxmf7u+iUVXplX702TsCkNAeYUeCuqr SF6A== X-Gm-Message-State: AOAM532KLq9ZNd4+HphJ4q+eXEc4+VBAA/swFkUA00I/2Ag6Eu2FMMtO IprAPO+qDstguu50twUCcYiHmRUJ/Q2qiv4ikO3tw23zveM= X-Google-Smtp-Source: ABdhPJygFsj5PNxIJIXmh0Tj90jbe4zZkAdLpYN8zARSvw2ivmR0Yy3PtZEbmpiHCFBDZmecypm6nV0/IGUS+A9XiRo= X-Received: by 2002:a5d:8417:: with SMTP id i23mr27005570ion.132.1595325455176; Tue, 21 Jul 2020 02:57:35 -0700 (PDT) MIME-Version: 1.0 From: Jesse Wang Date: Tue, 21 Jul 2020 17:57:24 +0800 Message-ID: Subject: Python and Java interoperability To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="0000000000000e970205aaf0a721" --0000000000000e970205aaf0a721 Content-Type: text/plain; charset="UTF-8" Hi, I want to have a Java process read the content of DataFrames produced by a Python process. The Java and Python processes run on different hosts. The solution I can think of is to have the Python process serialize the DataFrame and save it to redis, and have the Java process parse the data. The solution I find serializes the DataFrame to 'pybytes': (from https://stackoverflow.com/questions/57949871/how-to-set-get-pandas-dataframes-into-redis-using-pyarrow ) ``` import pandas as pd import pyarrow as paimport redis df=pd.DataFrame({'A':[1,2,3]}) r = redis.Redis(host='localhost', port=6379, db=0) context = pa.default_serialization_context() r.set("key", context.serialize(df).to_buffer().to_pybytes()) context.deserialize(r.get("key")) A0 11 22 3 ``` I wonder if this serialized 'pybytes' can be parsed at the Java end? If not, how can I achieve this properly? Thanks! -- Best Regards, Jiaxing Wang --0000000000000e970205aaf0a721 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,
I want to have a Java process read the conte= nt of DataFrames produced by a Python process. The Java and Python processe= s run=C2=A0on different hosts.

The solution I ca= n think of is to have the Python process serialize the DataFrame and save i= t to redis, and have the Java process parse the data.

<= div>The solution I find serializes the DataFrame to 'pybytes':
```
=C2=A0 =C2=A0import<= /span>=C2=A0pandas=C2=A0as=C2=A0pd
import pyarrow as pa
impo=
rt r=
edis

df=
=3Dp=
d.DataFrame({'A':[1,2,3]})
r =
=3D =
redis.Redis(host=3D'localhost', port=3D6379, db=3D0)

context =3D pa.d=
efault_serialization_context()
r.set("key&=
quot;, c=
ontext.s=
erialize(df)=
.to_=
buffer().to_pybytes())
context.=
deserialize(r.ge=
t("=
key"))
   A
0  1<=
span style=3D"margin:0px;padding:0px;border:0px;font-style:inherit;font-var=
iant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;f=
ont-family:inherit;vertical-align:baseline;box-sizing:inherit">
1  2<=
span style=3D"margin:0px;padding:0px;border:0px;font-style:inherit;font-var=
iant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;f=
ont-family:inherit;vertical-align:baseline;box-sizing:inherit">
2  3<=
/code>
```

I wonder if this serial= ized 'pybytes' can be parsed at the Java end? If not, how can I ach= ieve this properly?

Thanks!

--

Best Regards,
Jiaxing Wang
=C2=A0=
--0000000000000e970205aaf0a721--