arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@apache.org>
Subject Re: [Java] Flight fails to handle list vector
Date Mon, 15 Feb 2021 18:33:08 GMT
I think you need to call setPosition(0) before you start writing the list.
(This is from memory when I wrote the code 6-7 years ago so I may be off.)

On Sun, Feb 14, 2021 at 6:20 PM John Peterson <donttrytocontactme2@gmail.com>
wrote:

> Hi Bryan,
>
> This is the stacktrace I get:
>
> java.lang.IndexOutOfBoundsException: index: 0, length: 4 (expected:
> range(0, 0))
> at org.apache.arrow.memory.ArrowBuf.checkIndexD(ArrowBuf.java:318)
> at org.apache.arrow.memory.ArrowBuf.chk(ArrowBuf.java:305)
> at org.apache.arrow.memory.ArrowBuf.getInt(ArrowBuf.java:424)
> at
> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:97)
> at
> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:45)
> at
> org.apache.arrow.vector.BaseVariableWidthVector.accept(BaseVariableWidthVector.java:1402)
> at
> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:233)
> at
> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:45)
> at org.apache.arrow.vector.complex.ListVector.accept(ListVector.java:449)
> at
> org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:67)
> at
> org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:81)
>
> Thanks for your help.
>
> On Thu, Jan 14, 2021 at 2:23 PM Bryan Cutler <cutlerb@gmail.com> wrote:
>
>> Hi John, could you include the error with stacktrace?
>>
>> On Sat, Jan 9, 2021 at 9:34 PM John Peterson <
>> donttrytocontactme2@gmail.com> wrote:
>>
>>> I believe I'm running into a bug with Flight but I'd like to confirm and
>>> get some advice on a potential fix. I'm not sure where to look or what
>>> could be causing it.
>>>
>>> The code in question simply uploads a one-element List<Integer> to the
>>> example server, fetches it from the server, and attempts to append the data
>>> from the server to a new VectorSchemaRoot. It fails in the same way
>>> regardless of whether or not I construct a VectorSchemaRoot instance.
>>>
>>> Likewise, the data from the server can't be written out with the JSON
>>> writer, it'll fail in the same way. However, changing the data from a
>>> ListVector to an IntVector causes it to succeed.
>>>
>>> Any help would be appreciated.
>>>
>>> Thanks,
>>> John
>>>
>>> Code in question:
>>> // Set up the server and client
>>> BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
>>> Location l = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, 12233);
>>> ExampleFlightServer server = new ExampleFlightServer(allocator, l);
>>> server.start();
>>> FlightClient client = FlightClient.builder(allocator, l).build();
>>>
>>> // Write a one-element List<Integer>
>>> ListVector listVector = ListVector.empty("list", allocator);
>>> UnionListWriter writer = listVector.getWriter();
>>> writer.startList();
>>> writer.integer().writeInt(1);
>>> writer.endList();
>>> writer.setValueCount(1);
>>>
>>> // Send that data to the server
>>> VectorSchemaRoot root = VectorSchemaRoot.of(listVector);
>>> ClientStreamListener listener =
>>> client.startPut(FlightDescriptor.path("test"), root, new
>>> AsyncPutListener());
>>> root.setRowCount(1);
>>> listener.putNext();
>>> root.clear();
>>> listener.completed();
>>>
>>> // wait for ack to avoid memory leaks.
>>> listener.getResult();
>>>
>>> // Attempt to read it back
>>> FlightInfo info = client.getInfo(FlightDescriptor.path("test"));
>>> try (final FlightStream stream =
>>> client.getStream(info.getEndpoints().get(0).getTicket())) {
>>>   VectorSchemaRoot newRoot = stream.getRoot();
>>>   while (stream.next()) {
>>>     // Copying into an entirely new VectorSchemaRoot fails
>>>     try {
>>>       ListVector newList = ListVector.empty("list", allocator);
>>>
>>> newList.addOrGetVector(FieldType.nullable(Types.MinorType.INT.getType()));
>>>       VectorSchemaRoot copyRoot = VectorSchemaRoot.of(newList);
>>>       VectorSchemaRootAppender.append(copyRoot, newRoot);
>>>     } catch (IndexOutOfBoundsException e) {
>>>       System.err.println("Expected IOOBE caught");
>>>     }
>>>
>>>     // The same is true if we try to copy the data from the server to
>>> our VectorSchemaRoot
>>>     try {
>>>       VectorSchemaRootAppender.append(root, newRoot);
>>>     } catch (IndexOutOfBoundsException e) {
>>>       System.err.println("Expected IOOBE caught again");
>>>       throw e;
>>>     }
>>>
>>>     root.clear();
>>>     newRoot.clear();
>>>   }
>>> }
>>>
>>

Mime
View raw message