From user-return-908-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Wed Jan 13 18:57:16 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 67C2918066D for ; Wed, 13 Jan 2021 19:57:16 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 9200B4581E for ; Wed, 13 Jan 2021 18:57:15 +0000 (UTC) Received: (qmail 25209 invoked by uid 500); 13 Jan 2021 18:57:14 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 25199 invoked by uid 99); 13 Jan 2021 18:57:14 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2021 18:57:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 2B7861FF3A6 for ; Wed, 13 Jan 2021 18:57:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: -0.002 X-Spam-Level: X-Spam-Status: No, score=-0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id BA7qF0TT8_TN for ; Wed, 13 Jan 2021 18:57:13 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.219.54; helo=mail-qv1-f54.google.com; envelope-from=partha.dutta@gmail.com; receiver= Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 84367BD6E9 for ; Wed, 13 Jan 2021 18:57:13 +0000 (UTC) Received: by mail-qv1-f54.google.com with SMTP id l7so1263495qvt.4 for ; Wed, 13 Jan 2021 10:57:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=ymPVpngJrB8JkRHXehF3wy82yPVvVXWRF/eMSEdu2IQ=; b=CAMZ4+x7OgtoqRMnXxg/JZAXiHvP0tH3ZIE9eP4qGajnBWek26apVVy60Pt+B+mvG9 sLMIZKnptB3JlHxZRicyj//A24Lx+JVA/PtY4DU61Jb1UgO5+9m2iUJhxVCFMIZBJZm6 apo/eP+2ci/jmGKfhQvTfubMMzHKlsLGiNuDHQtcLB9OMfGHrvCdZ0iQ/JpV5gm8zBYJ oYeZcu79LSFKZWS8uFh7GH+djmMnBXni7DY8STIwoKQ072vyv6kmc2V5rhSQpZk950oJ dIbABuiFV5U19BMTO5tsG1MWVVuydJv/6MvmBTX58ZovSFp2onGPoixRoRitAkbzq2gb oCpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=ymPVpngJrB8JkRHXehF3wy82yPVvVXWRF/eMSEdu2IQ=; b=qg5wdfvkpPS0964+NNgxusnQKbkhwabDkXzjw5+zGOXhSoGYJjZYbapTt44fz12ST6 7jWICC7GRzTBxpgi5D/tG9bifil1RdqDftgtFiiAj9B3eKEYEMZmRG3ZuMEyhHi01h8T XRkKw0p97epXW4YM4YhY+8DWcl9dBLGSYMgSPeiNhzv7Arm0jbZ3A5pt5Z185tUswgrJ M9qE+NWCBfF/jQBHK3iADeeCBooOzpm9JSh0z2XViH06Js6iP5xO2IKDhK26fesFe5Vo zmwhJWv6ubuVfmErQy5bljckH7uYEtWM2K4xuo4Q++ZlTDF7K4jSFiQvzcbgaquv0ret neIg== X-Gm-Message-State: AOAM53204BfWPRf7qTuXfX+yiAZXkwF+1Gl2DkoH8KmCXoimkX/4v8Px I6FzscVCek1qWmfb6HCJwVQUx8eqQ8f72WwRCSFDz8YQ4lIWaw== X-Google-Smtp-Source: ABdhPJyw4JQhCpXNFZ9FUjs1G32O2XhRt80JBEJDzJkBZw3PVataXdWTRFicAg6Ml/dIR20Hoyhd6WMF2cEkaILzJmU= X-Received: by 2002:a0c:e9c2:: with SMTP id q2mr3564585qvo.1.1610564226544; Wed, 13 Jan 2021 10:57:06 -0800 (PST) MIME-Version: 1.0 From: PARTHA DUTTA Date: Wed, 13 Jan 2021 13:56:55 -0500 Message-ID: Subject: [Python] Possible to filter member of struct field? To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="0000000000009c792d05b8ccb4fd" --0000000000009c792d05b8ccb4fd Content-Type: text/plain; charset="UTF-8" I have a Parquet file which has a field defined as a struct: workEmail: struct child 0, address: string -- field metadata -- PARQUET:field_id: '13' -- field metadata -- PARQUET:field_id: '1' I am trying to write a filter as a DNF to query a specific value for workEmail.address but pyarrow does not seem to accept the DNF: tbl = pyarrow.parquet.read_table(filename, use_legacy_dataset=False, columns=["workEmail"], filters=[("workEmail.address", "=", "some@one.com")]) Is this supported? If not, any other workarounds? -- Partha Dutta partha.dutta@gmail.com --0000000000009c792d05b8ccb4fd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I have a Parquet file which has a field defined as a struc= t:
workEmail: struct<address: string>
=C2=A0 child 0, address:= string
=C2=A0 =C2=A0 -- field metadata --
=C2=A0 =C2=A0 PARQUET:fiel= d_id: '13'
=C2=A0 -- field metadata --
=C2=A0 PARQUET:field_i= d: '1'

I am trying to write a filter as a = DNF to query a specific value for workEmail.address but pyarrow does not se= em to accept the DNF:

tbl =3D pyarrow.parquet.read= _table(filename, use_legacy_dataset=3DFalse, columns=3D["workEmail&quo= t;], filters=3D[("workEmail.address", "=3D", "some@one.com")])

Is this supported? If not, any other work= arounds?

--
Partha Dutta
partha.dutta@gmail.com
--0000000000009c792d05b8ccb4fd--