From user-return-272-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Thu Jan 9 00:32:27 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id B72D7180607 for ; Thu, 9 Jan 2020 01:32:26 +0100 (CET) Received: (qmail 73890 invoked by uid 500); 9 Jan 2020 00:32:26 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 73880 invoked by uid 99); 9 Jan 2020 00:32:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2020 00:32:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A0DCC1A4300 for ; Thu, 9 Jan 2020 00:32:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id qyQ3kP04I-dT for ; Thu, 9 Jan 2020 00:32:23 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.217.50; helo=mail-vs1-f50.google.com; envelope-from=emkornfield@gmail.com; receiver= Received: from mail-vs1-f50.google.com (mail-vs1-f50.google.com [209.85.217.50]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id BF623BC563 for ; Thu, 9 Jan 2020 00:32:22 +0000 (UTC) Received: by mail-vs1-f50.google.com with SMTP id v12so3129189vsv.5 for ; Wed, 08 Jan 2020 16:32:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to; bh=yGpyMvgPCyAVUDZ0H3oKvY2qaaxocESskqZ3jj+7Jw4=; b=Ja7wMf6Fl0I0uKtGIlXbEUhIO1RYmccGCkRshOX1j5cvWUPvc1fX5bGCoPqeD6YRqb vntFe2bv16/3FTrf/Jr9oBlMw0KpHcIwrp06Qp0v9B4fnFL1JgJQhMSP+Z4xAPpHu2UU 0J6aE/EmIjWMI7H0VoTzJK7CsPgmbzSe0WVv6/sde9bd1Pk7la7hHl5DW+r5XN6SB50S LOSIUfKY+z7xxYBhXCVq2RNRjCQBJmBr0SpmSnUExBPyDtJpZ1xooK1uF9zJD3PR0Uo1 JQ5txZBF8ikeswFf86Sx28071AdCOxE2QqDja0zVQORtUzsklb2uxAx5g27PNsNKO1rW OlLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to; bh=yGpyMvgPCyAVUDZ0H3oKvY2qaaxocESskqZ3jj+7Jw4=; b=cEqGzVLP33Re+RDM7744q+bwbptqFpWa/cfQaeeUeXWCA1ZFWrpL+qh6k3H+S4Tknr CK9zrAKiAovEDJL9pR5YmyK+S0c1H6d4hqsJZ7SZl/RgManODnrO5hAYTT2NhBETGm9D l/RqLF4eL+qdAAZRKHl+8jpqRLOhadYBB+HUzu/vm1GoE1PibNxfUlNItw5fljG0kr7V 0kLMUMBikcayomdbx73OKpjDgDHAndfymx10E2qppJmf9rxowGcJFA/TYL36OcP+dHNX nsy7zx9nyP5pYJKrmWFOOD986DsbBz+7ZGRrGVRvXKaY9yXtvaqUByY/9eVtKJgkdjJ+ +8xw== X-Gm-Message-State: APjAAAV1I5/pW9S715lw48f/IP3KIfysB81nTBeSlAv3zg5bH+7nXz8z O1yDFbUZQJT/1EpiaIbjSPydXsSqvu3iSRGRprmj8A== X-Google-Smtp-Source: APXvYqxZMUpRHobWFa+fcqevXmErhS0//UUzMEpArBLqjr90EoX92u3w/oN70F0Z7SsiGseTLWuPZtJ9ciiYgHTPoqs= X-Received: by 2002:a67:d39a:: with SMTP id b26mr5105293vsj.119.1578529936346; Wed, 08 Jan 2020 16:32:16 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: Reply-To: emkornfield@gmail.com From: Micah Kornfield Date: Wed, 8 Jan 2020 16:32:05 -0800 Message-ID: Subject: Re: Implementation of Arrow table to Parquet File Writer To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="0000000000001f91d4059baa2436" --0000000000001f91d4059baa2436 Content-Type: text/plain; charset="UTF-8" There is not yet anything checked in in Arrow. There is an open PR to wrap the C++ parquet writer via JNI, however the C++ implementation does not yet supported nested columns. On Thu, Jan 2, 2020 at 3:06 AM saurabh pratap singh wrote: > forgot to mention using Java > > On Thu, Jan 2, 2020 at 3:41 PM saurabh pratap singh < > saurabh.cse16@gmail.com> wrote: > >> Hi >> >> I wanted to know whether there is a support/library available for >> writing arrow tables as parquet files. >> Meanwhile I tried writing my own converter where I am using >> SchemaConverter provided by arrow (to convert arrow schema to parquet ) >> Then Converting Arrow table to Group(ParquetExample Group reader/writer as >> a reference from parquet-mr) and dump as parquet .This works for >> primitive types without any issues but for nested types it will be little >> complicated so wanted to know if anything like this already exists or >> planned in near future . >> >> Thanks in advance.Please let me know if some other information is >> required from my side. >> >> --0000000000001f91d4059baa2436 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
There is not yet anything checked in in Arrow.=C2=A0 There= is an open PR to wrap the C++ parquet writer via JNI, however the C++ impl= ementation does not yet supported nested columns.

On Thu, Jan 2, 2020 at 3:0= 6 AM saurabh pratap singh <sa= urabh.cse16@gmail.com> wrote:
forgot to mention using Java
On Thu, J= an 2, 2020 at 3:41 PM saurabh pratap singh <saurabh.cse16@gmail.com> wrote:
=
Hi= =C2=A0

I wanted to know whether=C2=A0there is a support/= library available for writing=C2=A0arrow tables as parquet files.
Meanwhile I tried writing my own converter where I am using SchemaConverte= r provided by arrow (to convert arrow schema to parquet ) Then Converting A= rrow table to Group(ParquetExample Group reader/writer as a reference from = parquet-mr) and dump as parquet .This works for primitive=C2=A0types withou= t any issues but for nested types it will be little complicated so wanted t= o know if anything like this already exists or planned in near future .

Thanks in advance.Please let me know if some other in= formation is required from my side.

--0000000000001f91d4059baa2436--