drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-6234) VarCharVector setValueCount can throw IndexOutOfBoundsException
Date Tue, 13 Mar 2018 01:00:56 GMT

    [ https://issues.apache.org/jira/browse/DRILL-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396335#comment-16396335

Paul Rogers commented on DRILL-6234:

[~timothyfarkas], the tricky bit here is that your proposal is not simply a bug fix; it is
a change of design. Now, the way that the value count works is decidedly odd: one would naively
think that each vector would know how much data it holds. The value count name itself is misleading;
all that call really does is set the end-of-buffer location for the underlying {{ByteBuf}}.
(This is, in turn, because a value vector is not like a database buffer, it is instead modeled
on a network serialization/deserialization buffer.)

The call in {{VarCharVector}} fails because we need to set two positions: the end of the offsets
and the end of the values. To get the end of the values, we index into the offsets to find
the position of the (non-existent) n+1st item, and use that as the length of the values.

If we've not written any values, than the offsets are all zero (if the vector has been resized,
or if this is the first allocation from direct memory) or filled with garbage (if this is
the initial vector allocation and the buffer has been reused from the free list in which case
we don't zero the vector.)

So, it turns out, *there is no way* to set the end position of the values vector until we
write data. This is obviously not true of a fixed vector (such as the offsets) since we know
the width of each value. But, it is very true of variable-width vectors (VarChar) or arrays.

Bottom line: find some other way to accomplish your goal. The Drill protocol expects the client
to keep track of the item count until the vector is finished. For example, that is one of
the tasks that the result set loader handles: keeping track of the top-level values, and the
counts of values in nested arrays of maps of arrays of...

> VarCharVector setValueCount can throw IndexOutOfBoundsException
> ---------------------------------------------------------------
>                 Key: DRILL-6234
>                 URL: https://issues.apache.org/jira/browse/DRILL-6234
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Timothy Farkas
>            Assignee: Timothy Farkas
>            Priority: Major
> Doing the following will throw an Index out of bounds exception.
> {code}
>       final VarCharVector vector = new VarCharVector(field, allocator);
>       vector.allocateNew();
>       vector.getMutator().setValueCount(100);
> {code}
> The expected behavior is to resize the array appropriately. If an index is uninitialized
you should not call get for that index.
> {code}
> 	at org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
> 	at org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
> 	at io.netty.buffer.DrillBuf.chk(DrillBuf.java:114)
> 	at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:484)
> 	at org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:432)
> 	at org.apache.drill.exec.vector.VarCharVector$Mutator.setValueCount(VarCharVector.java:729)
> 	at org.apache.drill.exec.vector.VarCharVectorTest.testExpandingNonEmptyVectorSetValueCount(VarCharVectorTest.java:97)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> 	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> 	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> 	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}

This message was sent by Atlassian JIRA

View raw message