From user-return-995-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Tue Feb 16 03:21:20 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 68B2F18064D for ; Tue, 16 Feb 2021 04:21:20 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id A41DB43592 for ; Tue, 16 Feb 2021 03:21:19 +0000 (UTC) Received: (qmail 65016 invoked by uid 500); 16 Feb 2021 03:21:17 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 64996 invoked by uid 99); 16 Feb 2021 03:21:17 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Feb 2021 03:21:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 3E20E1FF39A for ; Tue, 16 Feb 2021 03:21:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: 0.249 X-Spam-Level: X-Spam-Status: No, score=0.249 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id kE-14E--PZ39 for ; Tue, 16 Feb 2021 03:21:15 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.173; helo=mail-oi1-f173.google.com; envelope-from=donttrytocontactme2@gmail.com; receiver= Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 1C796BCC67 for ; Tue, 16 Feb 2021 03:21:15 +0000 (UTC) Received: by mail-oi1-f173.google.com with SMTP id l19so9862191oih.6 for ; Mon, 15 Feb 2021 19:21:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=EbPJw225glppVfcXBqXZTd8CUH9gdMlX4Tn+ES/XU/8=; b=jbuFYhe9xER1MVGijefQoDojdDLgyOGndob0h57zwlT4RovPb0ualI/aklKpw4KA8v SL32QdDyeM0lK8Wrgs5zpw/2cUktNIAnMsPwck8MDcUa2hckbJMIyZzxoxZ/1kTGx8gS dV7yldn7ZXww9ZXjxsP3QQHM/98dB0kv5wdLK+LosfCvznO1IezWQkdmqY43P5ccDW3/ aqWM3JMtESR6izutV1i0+kf8NloAHqaQAxMB9nLE2n6/hgbGy4EG476exIfSc6iE3kpg g1dy/Nmfl+FDZEdVyJ951cQgSV2PKvj/FmY+E7eqK16LGCjmNUve9mZiuAMKcyg9i9bv VHjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=EbPJw225glppVfcXBqXZTd8CUH9gdMlX4Tn+ES/XU/8=; b=HRVB5M/5ZpMro5XZlsEXZDLtk895BDiVTdPgRc331SJUiGBZdoC+B+vHqEr9xH3sW6 kiFz8qWC5JynppC2A/SOA2bENm3FjPDW7szVGDBYBEK6Z98GWWE+MbDxgM6OHm0T+Cus t4BgiXBiL2c79LiEVFH1KkJoqt5VZmMBUvKUDa0eM6sH6XvblFY05YiTM5H+LlU7avqY iU2Z75AJXsjRlqA6FhDOd4nrSPRKRZ2neXBPOn5PfCwFSZrzcLEnOEabJSQvA6AyZE/x 0/C37ZV6BpxS+0PMBEk4zwgtQ7P/PQX216u7OUkvQnUSOnOuJ4NDD48KiVXoZg7Vl8KD k0Ww== X-Gm-Message-State: AOAM533eoqhP5UlFAQgDsPF5gSRC0Rfjhj+Lfc3v1YsqY1KDqLe+Fizb nnSgjg61SG1eE0dJhBaGEf2j59QiREu0CuJaChhv9koAEw== X-Google-Smtp-Source: ABdhPJwrGTc9RLbZ6KRLUJUTOqATbd8XIEJE2caE+kuvXjPeYurmrmJRzO9CQjMn3QggGZquGNOqSvZ48C1yitTlrqc= X-Received: by 2002:aca:f383:: with SMTP id r125mr1366164oih.18.1613445668302; Mon, 15 Feb 2021 19:21:08 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: John Peterson Date: Mon, 15 Feb 2021 22:21:02 -0500 Message-ID: Subject: Re: [Java] Flight fails to handle list vector To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000ec714405bb6b97f7" --000000000000ec714405bb6b97f7 Content-Type: text/plain; charset="UTF-8" Appreciate the help Jacques. Unfortunately calling setPosition(0) on the writer for the list did not solve it. I put the entirety of the code up on pastebin so it should be an easy copy/paste if anybody else wants to try to reproduce it. I suppose it could also be a bug in VectorAppender, but again I'm not sure if the error is in my code or in Arrow. https://pastebin.com/vwvnYY40 Thanks in advance. On Mon, Feb 15, 2021 at 1:33 PM Jacques Nadeau wrote: > I think you need to call setPosition(0) before you start writing the list. > (This is from memory when I wrote the code 6-7 years ago so I may be off.) > > On Sun, Feb 14, 2021 at 6:20 PM John Peterson < > donttrytocontactme2@gmail.com> wrote: > >> Hi Bryan, >> >> This is the stacktrace I get: >> >> java.lang.IndexOutOfBoundsException: index: 0, length: 4 (expected: >> range(0, 0)) >> at org.apache.arrow.memory.ArrowBuf.checkIndexD(ArrowBuf.java:318) >> at org.apache.arrow.memory.ArrowBuf.chk(ArrowBuf.java:305) >> at org.apache.arrow.memory.ArrowBuf.getInt(ArrowBuf.java:424) >> at >> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:97) >> at >> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:45) >> at >> org.apache.arrow.vector.BaseVariableWidthVector.accept(BaseVariableWidthVector.java:1402) >> at >> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:233) >> at >> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:45) >> at org.apache.arrow.vector.complex.ListVector.accept(ListVector.java:449) >> at >> org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:67) >> at >> org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:81) >> >> Thanks for your help. >> >> On Thu, Jan 14, 2021 at 2:23 PM Bryan Cutler wrote: >> >>> Hi John, could you include the error with stacktrace? >>> >>> On Sat, Jan 9, 2021 at 9:34 PM John Peterson < >>> donttrytocontactme2@gmail.com> wrote: >>> >>>> I believe I'm running into a bug with Flight but I'd like to confirm >>>> and get some advice on a potential fix. I'm not sure where to look or what >>>> could be causing it. >>>> >>>> The code in question simply uploads a one-element List to the >>>> example server, fetches it from the server, and attempts to append the data >>>> from the server to a new VectorSchemaRoot. It fails in the same way >>>> regardless of whether or not I construct a VectorSchemaRoot instance. >>>> >>>> Likewise, the data from the server can't be written out with the JSON >>>> writer, it'll fail in the same way. However, changing the data from a >>>> ListVector to an IntVector causes it to succeed. >>>> >>>> Any help would be appreciated. >>>> >>>> Thanks, >>>> John >>>> >>>> Code in question: >>>> // Set up the server and client >>>> BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE); >>>> Location l = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, 12233); >>>> ExampleFlightServer server = new ExampleFlightServer(allocator, l); >>>> server.start(); >>>> FlightClient client = FlightClient.builder(allocator, l).build(); >>>> >>>> // Write a one-element List >>>> ListVector listVector = ListVector.empty("list", allocator); >>>> UnionListWriter writer = listVector.getWriter(); >>>> writer.startList(); >>>> writer.integer().writeInt(1); >>>> writer.endList(); >>>> writer.setValueCount(1); >>>> >>>> // Send that data to the server >>>> VectorSchemaRoot root = VectorSchemaRoot.of(listVector); >>>> ClientStreamListener listener = >>>> client.startPut(FlightDescriptor.path("test"), root, new >>>> AsyncPutListener()); >>>> root.setRowCount(1); >>>> listener.putNext(); >>>> root.clear(); >>>> listener.completed(); >>>> >>>> // wait for ack to avoid memory leaks. >>>> listener.getResult(); >>>> >>>> // Attempt to read it back >>>> FlightInfo info = client.getInfo(FlightDescriptor.path("test")); >>>> try (final FlightStream stream = >>>> client.getStream(info.getEndpoints().get(0).getTicket())) { >>>> VectorSchemaRoot newRoot = stream.getRoot(); >>>> while (stream.next()) { >>>> // Copying into an entirely new VectorSchemaRoot fails >>>> try { >>>> ListVector newList = ListVector.empty("list", allocator); >>>> >>>> newList.addOrGetVector(FieldType.nullable(Types.MinorType.INT.getType())); >>>> VectorSchemaRoot copyRoot = VectorSchemaRoot.of(newList); >>>> VectorSchemaRootAppender.append(copyRoot, newRoot); >>>> } catch (IndexOutOfBoundsException e) { >>>> System.err.println("Expected IOOBE caught"); >>>> } >>>> >>>> // The same is true if we try to copy the data from the server to >>>> our VectorSchemaRoot >>>> try { >>>> VectorSchemaRootAppender.append(root, newRoot); >>>> } catch (IndexOutOfBoundsException e) { >>>> System.err.println("Expected IOOBE caught again"); >>>> throw e; >>>> } >>>> >>>> root.clear(); >>>> newRoot.clear(); >>>> } >>>> } >>>> >>> --000000000000ec714405bb6b97f7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Appreciate the help Jacques. Unfortunately calling setPosi= tion(0) on the writer for the list did not solve it.

I p= ut the entirety of the code up on pastebin so it should be an easy copy/pas= te if anybody else wants to try to reproduce it. I suppose it could also be= a bug in VectorAppender, but again I'm not sure if the error is in my = code or in Arrow.


Th= anks in advance.


<= div dir=3D"ltr" class=3D"gmail_attr">On Mon, Feb 15, 2021 at 1:33 PM Jacque= s Nadeau <jacques@apache.org&g= t; wrote:
I think you need to call setPosition(0) before you start writing= the list. (This is from memory when I wrote the code 6-7 years ago so I ma= y be off.)

On Sun, Feb 14, 2021 at 6:20 PM John Peterson <donttrytocontac= tme2@gmail.com> wrote:
Hi Bryan,

This is the stacktrace=C2=A0I get:

java.lang.Ind= exOutOfBoundsException: index: 0, length: 4 (expected: range(0, 0))
at = org.apache.arrow.memory.ArrowBuf.checkIndexD(ArrowBuf.java:318)
at org.= apache.arrow.memory.ArrowBuf.chk(ArrowBuf.java:305)
at org.apache.arrow= .memory.ArrowBuf.getInt(ArrowBuf.java:424)
at org.apache.arrow.vector.u= til.VectorAppender.visit(VectorAppender.java:97)
at org.apache.arrow.ve= ctor.util.VectorAppender.visit(VectorAppender.java:45)
at org.apache.ar= row.vector.BaseVariableWidthVector.accept(BaseVariableWidthVector.java:1402= )
at org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.j= ava:233)
at org.apache.arrow.vector.util.VectorAppender.visit(VectorApp= ender.java:45)
at org.apache.arrow.vector.complex.ListVector.accept(Lis= tVector.java:449)
at org.apache.arrow.vector.util.VectorSchemaRootAppen= der.append(VectorSchemaRootAppender.java:67)
at org.apache.arrow.vector= .util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:81)
=

Thanks for your help.

On Thu, Jan 14, 2= 021 at 2:23 PM Bryan Cutler <cutlerb@gmail.com> wrote:
Hi John, could you include the error with stacktrace?
<= /div>

On Sat, Jan 9, 2021 at 9:34 PM John Peterson <donttrytocontactme2@gmail.com= > wrote:
=
I believe I'm running into a bug with Flight but = I'd like to confirm and get some advice on a potential fix. I'm not= sure where to look or what could be causing it.=C2=A0

=
The code in question simply uploads a one-element List<Integer> = to the example server, fetches it from the server, and attempts to append t= he data from the server to a new VectorSchemaRoot. It fails in the same way= regardless of whether or not I construct a VectorSchemaRoot instance.=C2= =A0

Likewise, the data from the server can't b= e written out with the JSON writer, it'll fail in the same way. However= , changing the data from a ListVector to an IntVector causes it to succeed.= =C2=A0

Any help would be appreciated.
Thanks,
John

Code in questi= on:
// Set up the server and client
BufferAllocator allocator = =3D new RootAllocator(Long.MAX_VALUE);
Location l =3D Location.forGrpcIn= secure(FlightTestUtil.LOCALHOST, 12233);
ExampleFlightServer server =3D = new ExampleFlightServer(allocator, l);
server.start();
FlightClient c= lient =3D FlightClient.builder(allocator, l).build();

// Write a one= -element List<Integer>
ListVector listVector =3D ListVector.empty(= "list", allocator);
UnionListWriter writer =3D listVector.getW= riter();
writer.startList();
writer.integer().writeInt(1);
writer.= endList();
writer.setValueCount(1);

// Send that data to the serv= er
VectorSchemaRoot root =3D VectorSchemaRoot.of(listVector);
ClientS= treamListener listener =3D client.startPut(FlightDescriptor.path("test= "), root, new AsyncPutListener());
root.setRowCount(1);
listener= .putNext();
root.clear();
listener.completed();

// wait for ac= k to avoid memory leaks.
listener.getResult();

// Attempt to read= it back
FlightInfo info =3D client.getInfo(FlightDescriptor.path("= test"));
try (final FlightStream stream =3D client.getStream(info.g= etEndpoints().get(0).getTicket())) {
=C2=A0 VectorSchemaRoot newRoot =3D= stream.getRoot();
=C2=A0 while (stream.next()) {
=C2=A0 =C2=A0 // Co= pying into an entirely new VectorSchemaRoot fails
=C2=A0 =C2=A0 try {=C2=A0 =C2=A0 =C2=A0 ListVector newList =3D ListVector.empty("list&qu= ot;, allocator);
=C2=A0 =C2=A0 =C2=A0 newList.addOrGetVector(FieldType.n= ullable(Types.MinorType.INT.getType()));
=C2=A0 =C2=A0 =C2=A0 VectorSche= maRoot copyRoot =3D VectorSchemaRoot.of(newList);
=C2=A0 =C2=A0 =C2=A0 V= ectorSchemaRootAppender.append(copyRoot, newRoot);
=C2=A0 =C2=A0 } catch= (IndexOutOfBoundsException e) {
=C2=A0 =C2=A0 =C2=A0 System.err.println= ("Expected IOOBE caught");
=C2=A0 =C2=A0 }

=C2=A0 =C2= =A0 // The same is true if we try to copy the data from the server to our V= ectorSchemaRoot
=C2=A0 =C2=A0 try {
=C2=A0 =C2=A0 =C2=A0 VectorSchema= RootAppender.append(root, newRoot);
=C2=A0 =C2=A0 } catch (IndexOutOfBou= ndsException e) {
=C2=A0 =C2=A0 =C2=A0 System.err.println("Expected= IOOBE caught again");
=C2=A0 =C2=A0 =C2=A0 throw e;
=C2=A0 =C2= =A0 }

=C2=A0 =C2=A0 root.clear();
=C2=A0 =C2=A0 newRoot.clear();<= br>=C2=A0 }
}
--000000000000ec714405bb6b97f7--