arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@apache.org>
Subject Re: [Java] Flight fails to handle list vector
Date Tue, 16 Feb 2021 17:31:02 GMT
Hey John, a brief review of your code makes me wonder if the problem may be
associated with VectorSchemaRootAppender. Can you try your test without
that. Basically, once you get a batch of data back, inspect it to see that
you have your values. VectorSchemaRootAppender is new code that I haven't
reviewed and I'm wondering if it isn't handling reference counting
correctly.

The exception you're seeing is most frequently associated with what could
be thought of as a NPE for the memory backing a vector. When a vectors are
like a container. The design was built so a vector has batches stream
through it. When no buffer is available, rather than setting the buffer to
null, we set it to the empty buffer (which is of zero length). If you try
to do something with the vector when it is empty. In this case, my guess is
you are trying to read the start offset for the first item in a list e.g.
the first four bytes [0..4) of the vector but the vector is only 0 bytes in
length (thus the exception).

On Mon, Feb 15, 2021 at 7:21 PM John Peterson <donttrytocontactme2@gmail.com>
wrote:

> Appreciate the help Jacques. Unfortunately calling setPosition(0) on the
> writer for the list did not solve it.
>
> I put the entirety of the code up on pastebin so it should be an easy
> copy/paste if anybody else wants to try to reproduce it. I suppose it could
> also be a bug in VectorAppender, but again I'm not sure if the error is in
> my code or in Arrow.
>
> https://pastebin.com/vwvnYY40
>
> Thanks in advance.
>
>
> On Mon, Feb 15, 2021 at 1:33 PM Jacques Nadeau <jacques@apache.org> wrote:
>
>> I think you need to call setPosition(0) before you start writing the
>> list. (This is from memory when I wrote the code 6-7 years ago so I may be
>> off.)
>>
>> On Sun, Feb 14, 2021 at 6:20 PM John Peterson <
>> donttrytocontactme2@gmail.com> wrote:
>>
>>> Hi Bryan,
>>>
>>> This is the stacktrace I get:
>>>
>>> java.lang.IndexOutOfBoundsException: index: 0, length: 4 (expected:
>>> range(0, 0))
>>> at org.apache.arrow.memory.ArrowBuf.checkIndexD(ArrowBuf.java:318)
>>> at org.apache.arrow.memory.ArrowBuf.chk(ArrowBuf.java:305)
>>> at org.apache.arrow.memory.ArrowBuf.getInt(ArrowBuf.java:424)
>>> at
>>> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:97)
>>> at
>>> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:45)
>>> at
>>> org.apache.arrow.vector.BaseVariableWidthVector.accept(BaseVariableWidthVector.java:1402)
>>> at
>>> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:233)
>>> at
>>> org.apache.arrow.vector.util.VectorAppender.visit(VectorAppender.java:45)
>>> at org.apache.arrow.vector.complex.ListVector.accept(ListVector.java:449)
>>> at
>>> org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:67)
>>> at
>>> org.apache.arrow.vector.util.VectorSchemaRootAppender.append(VectorSchemaRootAppender.java:81)
>>>
>>> Thanks for your help.
>>>
>>> On Thu, Jan 14, 2021 at 2:23 PM Bryan Cutler <cutlerb@gmail.com> wrote:
>>>
>>>> Hi John, could you include the error with stacktrace?
>>>>
>>>> On Sat, Jan 9, 2021 at 9:34 PM John Peterson <
>>>> donttrytocontactme2@gmail.com> wrote:
>>>>
>>>>> I believe I'm running into a bug with Flight but I'd like to confirm
>>>>> and get some advice on a potential fix. I'm not sure where to look or
what
>>>>> could be causing it.
>>>>>
>>>>> The code in question simply uploads a one-element List<Integer>
to the
>>>>> example server, fetches it from the server, and attempts to append the
data
>>>>> from the server to a new VectorSchemaRoot. It fails in the same way
>>>>> regardless of whether or not I construct a VectorSchemaRoot instance.
>>>>>
>>>>> Likewise, the data from the server can't be written out with the JSON
>>>>> writer, it'll fail in the same way. However, changing the data from a
>>>>> ListVector to an IntVector causes it to succeed.
>>>>>
>>>>> Any help would be appreciated.
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> Code in question:
>>>>> // Set up the server and client
>>>>> BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
>>>>> Location l = Location.forGrpcInsecure(FlightTestUtil.LOCALHOST, 12233);
>>>>> ExampleFlightServer server = new ExampleFlightServer(allocator, l);
>>>>> server.start();
>>>>> FlightClient client = FlightClient.builder(allocator, l).build();
>>>>>
>>>>> // Write a one-element List<Integer>
>>>>> ListVector listVector = ListVector.empty("list", allocator);
>>>>> UnionListWriter writer = listVector.getWriter();
>>>>> writer.startList();
>>>>> writer.integer().writeInt(1);
>>>>> writer.endList();
>>>>> writer.setValueCount(1);
>>>>>
>>>>> // Send that data to the server
>>>>> VectorSchemaRoot root = VectorSchemaRoot.of(listVector);
>>>>> ClientStreamListener listener =
>>>>> client.startPut(FlightDescriptor.path("test"), root, new
>>>>> AsyncPutListener());
>>>>> root.setRowCount(1);
>>>>> listener.putNext();
>>>>> root.clear();
>>>>> listener.completed();
>>>>>
>>>>> // wait for ack to avoid memory leaks.
>>>>> listener.getResult();
>>>>>
>>>>> // Attempt to read it back
>>>>> FlightInfo info = client.getInfo(FlightDescriptor.path("test"));
>>>>> try (final FlightStream stream =
>>>>> client.getStream(info.getEndpoints().get(0).getTicket())) {
>>>>>   VectorSchemaRoot newRoot = stream.getRoot();
>>>>>   while (stream.next()) {
>>>>>     // Copying into an entirely new VectorSchemaRoot fails
>>>>>     try {
>>>>>       ListVector newList = ListVector.empty("list", allocator);
>>>>>
>>>>> newList.addOrGetVector(FieldType.nullable(Types.MinorType.INT.getType()));
>>>>>       VectorSchemaRoot copyRoot = VectorSchemaRoot.of(newList);
>>>>>       VectorSchemaRootAppender.append(copyRoot, newRoot);
>>>>>     } catch (IndexOutOfBoundsException e) {
>>>>>       System.err.println("Expected IOOBE caught");
>>>>>     }
>>>>>
>>>>>     // The same is true if we try to copy the data from the server to
>>>>> our VectorSchemaRoot
>>>>>     try {
>>>>>       VectorSchemaRootAppender.append(root, newRoot);
>>>>>     } catch (IndexOutOfBoundsException e) {
>>>>>       System.err.println("Expected IOOBE caught again");
>>>>>       throw e;
>>>>>     }
>>>>>
>>>>>     root.clear();
>>>>>     newRoot.clear();
>>>>>   }
>>>>> }
>>>>>
>>>>

Mime
View raw message