From user-return-775-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Fri Nov 13 06:42:40 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id E5CB3180658 for ; Fri, 13 Nov 2020 07:42:39 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 352E94A65E for ; Fri, 13 Nov 2020 06:42:39 +0000 (UTC) Received: (qmail 96083 invoked by uid 500); 13 Nov 2020 06:42:38 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 96073 invoked by uid 99); 13 Nov 2020 06:42:38 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Nov 2020 06:42:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 9B3D4BFD70 for ; Fri, 13 Nov 2020 06:42:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id rF_TZABxiwec for ; Fri, 13 Nov 2020 06:42:37 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::635; helo=mail-ej1-x635.google.com; envelope-from=emkornfield@gmail.com; receiver= Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id EED437F7BC for ; Fri, 13 Nov 2020 06:42:36 +0000 (UTC) Received: by mail-ej1-x635.google.com with SMTP id cw8so11719945ejb.8 for ; Thu, 12 Nov 2020 22:42:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to; bh=C3C7wXFQB0WErHkjS5pcLFCRNoTVXa/SUSbgZOwrdtg=; b=mUYXOk2TBunOI1l8JowJE+kILOZeo7darJoLgYK+rnRo2MA6dpC5GoYMKVNbDOTHum yzuxGX+KFhQaCRF+JrG/CmFchC/0zZjyjpGP+QdaY/7ovAR9nPin5diHjYdg+vs7T/PT 21boBTd0AyHAn0PuHQaGwsqSv2BBxCLCuyPnvHuDBaFJfvOPJcdJ9e/vtDZiWvR7ItRb YRrQkFAdBYDU8XqT8OGz/afN6TJAJ6ay33hGcAs8Ins5TZ9lqvfeJwy0mx8wAl6jqYn4 z/WjiI+tXbPN5NE1gA0bT/ngPl4nTMY/NMbxJmRtKYQpZ0XaLLoRjKdTng6qU4IF12Tg TrrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to; bh=C3C7wXFQB0WErHkjS5pcLFCRNoTVXa/SUSbgZOwrdtg=; b=VHtD5vbuugXNcnMe/6a5lkzkUMFQgmED1L6WpZfNX5fuvrw53CMqApyN/krAmPnDib ZCgwPyTkon/rvnqPlj/RXhxIU3ZHfVHJESjnxNKZZAij+a42z52sXklVFLl9ByiIlurc 93EtJ4lg6y/Otsxff7Ap5gweGFPEmWCO1cA/I0GKniJ3gWWdpJgeKaFCOVj/A728RFim 8/Im1s1uZJ/PTiY9H+uqvmkL9jyh8JuU0kSdI4DUsUCBs6QrSz2FzQaH4C6XnfGnNL/E FL39I4Ic5Lp5Syr/xrPHi9yX3BhtJu4XnSAJzBattOcshDIfs6R9pAW2v9/6NzfYaNkQ Sk3Q== X-Gm-Message-State: AOAM533pf3Q+UlvlLfeSkyCqJeXgvRhq4gW/mIkpRLVR9nY+iqf+MRmD PCCAa5RK1DrBsNUX/mj4B1rEHWATeS9gEqye6Cz7ECuNWH1U5Q== X-Google-Smtp-Source: ABdhPJwi+bEdLT5RJpOG6uhQ01Z6XlA6oP0uegAPFPz8ZxnzTk4T6FkQXjOrr5PV6UAa+SQE5EgzICxDV8W2KAPPjCY= X-Received: by 2002:a17:906:6d52:: with SMTP id a18mr593365ejt.224.1605249756153; Thu, 12 Nov 2020 22:42:36 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: Reply-To: emkornfield@gmail.com From: Micah Kornfield Date: Thu, 12 Nov 2020 22:42:25 -0800 Message-ID: Subject: Re: Arrow java implementation: Compatible IO streams. To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="0000000000007dcaba05b3f75515" --0000000000007dcaba05b3f75515 Content-Type: text/plain; charset="UTF-8" Hi Saloni, Sample code for how you are using the Arrow APIs are useful. Also how can I validate if the output generated by Arrow writer is in valid > format? Being able to read it back would be the best mechanism here? Are you think of unit tests or in production? Or something else? Thanks, Micah On Wed, Nov 4, 2020 at 12:43 AM Saloni Udani wrote: > Hello, > I have a use case where I want to write an arrow batch to my existing > output stream (custom stream extending java.io.OutputStream) and reading > from my existing input stream (custom stream extending > java.io.InputStream). I used ArrowStreamWriter and ArrowStreamReader but > on the reader side I get NPE while getting VectorSchemaRoot. > > Caused by: java.lang.NullPointerException: null > at > org.apache.arrow.vector.types.pojo.Field.mutateOriginalNameIfNeeded(Field.java:188) > at org.apache.arrow.vector.types.pojo.Field.convertField(Field.java:172) > at org.apache.arrow.vector.types.pojo.Schema.convertSchema(Schema.java:87) > at > org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeSchema(MessageSerializer.java:201) > at > org.apache.arrow.vector.ipc.ArrowStreamReader.readSchema(ArrowStreamReader.java:174) > at org.apache.arrow.vector.ipc.ArrowReader.initialize(ArrowReader.java:170) > at > org.apache.arrow.vector.ipc.ArrowReader.ensureInitialized(ArrowReader.java:161) > at > org.apache.arrow.vector.ipc.ArrowReader.getVectorSchemaRoot(ArrowReader.java:63) > > > > > Kindly guide what am I doing wrong? I believe this to be an issue of IO > streams. Is there any prerequisite to which IO streams can be used with > Arrow reader and writer? > > Also how can I validate if the output generated by Arrow writer is in > valid format? > > > Regards > Saloni Udani > --0000000000007dcaba05b3f75515 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Saloni,
Sample code for how you are using the Arrow= APIs are useful.=C2=A0=C2=A0

Also how can I validate if the output generated by= Arrow writer is in valid format?
Being able to read it ba= ck would be the best mechanism here?=C2=A0 Are you think of unit tests or i= n production?=C2=A0 Or something else?=C2=A0

Thank= s,
Micah

On Wed, Nov 4, 2020 at 12:43 AM Saloni Udani <saloniudani.tech@gmail.com&= gt; wrote:
Hello,
I have a use case where I want to wri= te an arrow batch to my existing output stream (custom stream extending jav= a.io.OutputStream)=C2=A0 and reading from my existing input stream (custom = stream extending java.io.InputStream).=C2=A0 I used ArrowStreamWriter and A= rrowStreamReader but on the reader side I get NPE while getting VectorSchem= aRoot.

Caused by: java.lang.NullPointerException: = null
at org.apache.arrow.vector.types.pojo.Field.mutateOriginalNameIfNe= eded(Field.java:188)
at org.apache.arrow.vector.types.pojo.Field.conver= tField(Field.java:172)
at org.apache.arrow.vector.types.pojo.Schema.con= vertSchema(Schema.java:87)
at org.apache.arrow.vector.ipc.message.Messa= geSerializer.deserializeSchema(MessageSerializer.java:201)
at org.apach= e.arrow.vector.ipc.ArrowStreamReader.readSchema(ArrowStreamReader.java:174)=
at org.apache.arrow.vector.ipc.ArrowReader.initialize(ArrowReader.java= :170)
at org.apache.arrow.vector.ipc.ArrowReader.ensureInitialized(Arro= wReader.java:161)
at org.apache.arrow.vector.ipc.ArrowReader.getVectorS= chemaRoot(ArrowReader.java:63)



=

Kindly guide what am I doing wrong? I believe thi= s to be an issue of IO streams. Is there any prerequisite to which IO strea= ms can be used with Arrow reader and writer?

Also how can= I validate if the output generated by Arrow writer is in valid format?


Regards
Saloni Udani
--0000000000007dcaba05b3f75515--