Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Message-ID: <5621C76B.9060505@maprtech.com>
Date: Fri, 16 Oct 2015 20:58:35 -0700
From: Daniel Barclay <dbarclay@maprtech.com>
Organization: MapR Technologies
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:36.0) Gecko/20100101 Firefox/36.0 SeaMonkey/2.33.1
MIME-Version: 1.0
To: Chris Westin <cwestin@maprtech.com>, dev@drill.apache.org
Subject: intermittent IndexOutOfBoundsException from uninitialized
 null-bits vector?
References: 
 <CANxDS=JDKZrKDVWHGrrbU3EOGrdWU-b2K+9ohKOxRQ4hKu8EgQ@mail.gmail.com>
In-Reply-To: 
 <CANxDS=JDKZrKDVWHGrrbU3EOGrdWU-b2K+9ohKOxRQ4hKu8EgQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------070006080004040604040008"

--------------070006080004040604040008
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

[With corrected track trace.]

Chris,

Actually, now I am seeing a non-deterministic IOOBE like yours (with leng=
th =3D 1).

Note in the call stack way below that it's coming from an isNull() method=
=2E  The isNull() method was called with an index of 0 when the top-level=
 vector container or whatever had one row.

*It looks like the subvector used to track null values didn't get filled =
in right. *(I can't tell yet if it also means that the value-printing cod=
e in the HBase test is missing something about a schema change.)

This is from an HBase query with "WHERE row_key =3D 'a2' or row_key betwe=
en 'b5' and 'b6'".  A batch of 2 rows resulting from the "between ..." pa=
rt comes first--*non-deterministically* (it was consistent for many runs,=
 but it switched just now in a run for adding the schemas to this message=
)--and then the batch of one row for the "=3D 'a2'" part seems messed up =
in the second column family (f2):

The first batch's schema is (note *f2*'s type):
BatchSchema [fields=3D[`row_key`(VARBINARY:REQUIRED)[`$offsets$`(UINT4:RE=
QUIRED)], `f`(MAP:REQUIRED)[`f`.`c1`(VARBINARY:OPTIONAL)[`$bits$`(UINT1:R=
EQUIRED), `f`.`c1`(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]], `f`=
=2E`c2`(VARBINARY:OPTIONAL)[`$bits$`(UINT1:REQUIRED), `f`.`c2`(VARBINARY:=
OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]], `f`.`c3`(VARBINARY:OPTIONAL)[`$b=
its$`(UINT1:REQUIRED), `f`.`c3`(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQ=
UIRED)]], `f`.`c4`(VARBINARY:OPTIONAL)[`$bits$`(UINT1:REQUIRED), `f`.`c4`=
(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]], `f`.`c5`(VARBINARY:OP=
TIONAL)[`$bits$`(UINT1:REQUIRED), `f`.`c5`(VARBINARY:OPTIONAL)[`$offsets$=
`(UINT4:REQUIRED)]], `f`.`c6`(VARBINARY:OPTIONAL)[`$bits$`(UINT1:REQUIRED=
), `f`.`c6`(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]], `f`.`c8`(V=
ARBINARY:OPTIONAL)[`$bits$`(UINT1:REQUIRED), `f`.`c8`(VARBINARY:OPTIONAL)=
[`$offsets$`(UINT4:REQUIRED)]]], `f2`(MAP:REQUIRED)[`f2`.`c1`(VARBINARY:O=
PTIONAL)[`$bits$`(UINT1:REQUIRED),=20
`f2`.`c1`(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]], `f2`.`c3`(VA=
RBINARY:OPTIONAL)[`$bits$`(UINT1:REQUIRED), `f2`.`c3`(VARBINARY:OPTIONAL)=
[`$offsets$`(UINT4:REQUIRED)]], `f2`.`c5`(VARBINARY:OPTIONAL)[`$bits$`(UI=
NT1:REQUIRED), `f2`.`c5`(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]=
], `f2`.`c7`(VARBINARY:OPTIONAL)[`$bits$`(UINT1:REQUIRED), `f2`.`c7`(VARB=
INARY:OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]], `f2`.`c9`(VARBINARY:OPTION=
AL)[`$bits$`(UINT1:REQUIRED), `f2`.`c9`(VARBINARY:OPTIONAL)[`$offsets$`(U=
INT4:REQUIRED)]]]], selectionVector=3DNONE]

The second batch's schema is (*note f2's type*):
BatchSchema [fields=3D[`row_key`(VARBINARY:REQUIRED)[`$offsets$`(UINT4:RE=
QUIRED)], `f`(MAP:REQUIRED)[`f`.`c1`(VARBINARY:OPTIONAL)[`$bits$`(UINT1:R=
EQUIRED), `f`.`c1`(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]], `f`=
=2E`c2`(VARBINARY:OPTIONAL)[`$bits$`(UINT1:REQUIRED), `f`.`c2`(VARBINARY:=
OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]], `f`.`c3`(VARBINARY:OPTIONAL)[`$b=
its$`(UINT1:REQUIRED), `f`.`c3`(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQ=
UIRED)]], `f`.`c4`(VARBINARY:OPTIONAL)[`$bits$`(UINT1:REQUIRED), `f`.`c4`=
(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]], `f`.`c5`(VARBINARY:OP=
TIONAL)[`$bits$`(UINT1:REQUIRED), `f`.`c5`(VARBINARY:OPTIONAL)[`$offsets$=
`(UINT4:REQUIRED)]], `f`.`c6`(VARBINARY:OPTIONAL)[`$bits$`(UINT1:REQUIRED=
), `f`.`c6`(VARBINARY:OPTIONAL)[`$offsets$`(UINT4:REQUIRED)]]], `f2`(INT:=
OPTIONAL)[`$bits$`(UINT1:REQUIRED), `f2`(INT:OPTIONAL)]], selectionVector=
=3DNONE]

*Is f2's type of **INT:OPTIONAL**correct?*

Why wouldn't f2's type be MAP:REQUIRED? Even if the HBase reader didn't s=
ee any HBase columns in HBase column family f2, doesn't it still know tha=
t f2 is a column family and shouldn't it still set Drill's f2 column to b=
e a map and not of type INT:OPTIONAL?

Stack trace at IndexOutOfBoundsException saying "index: 0, length: 1 (exp=
ected: range(0, 0))":
IndexOutOfBoundsException.<init>(String) line: 56
DrillBuf.checkIndexD(int, int) line: 189
DrillBuf.chk(int, int) line: 211
DrillBuf.getByte(int) line: 746
UInt1Vector$Accessor.get(int) line: 364
NullableVarBinaryVector$Accessor.isSet(int) line: 391
NullableVarBinaryVector$Accessor.isNull(int) line: 387
NullableVarBinaryVector$Accessor.getObject(int) line: 411
NullableVarBinaryVector$Accessor.getObject(int) line: 1
MapVector$Accessor.getObject(int) line: 313
VectorUtil.showVectorAccessibleContent(VectorAccessible, int[]) line: 167=

TestHBaseFilterPushDown(BaseTestQuery).printResult(List<QueryDataBatch>) =
line: 487
TestHBaseFilterPushDown(BaseHBaseTest).printResultAndVerifyRowCount(List<=
QueryDataBatch>, int) line: 95
TestHBaseFilterPushDown(BaseHBaseTest).runHBaseSQLVerifyCount(String, int=
) line: 91
TestHBaseFilterPushDown.testFilterPushDownOrRowKeyEqualRangePred() line: =
750
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not avai=
lable [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 606
FrameworkMethod$1.runReflectiveCall() line: 47
FrameworkMethod$1(ReflectiveCallable).run() line: 12
FrameworkMethod.invokeExplosively(Object, Object...) line: 44
JUnit4TestRunnerDecorator.executeTestMethod(FrameworkMethod, Object, Obje=
ct...) line: 120
JUnit4TestRunnerDecorator.invokeExplosively(FrameworkMethod, Object, Obje=
ct...) line: 65
MockFrameworkMethod.invokeExplosively(Invocation, Object, Object...) line=
: 29
GeneratedMethodAccessor133.invoke(Object, Object[]) line: not available
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 606
MethodReflection.invokeWithCheckedThrows(Object, Method, Object...) line:=
 95
MockMethodBridge.callMock(Object, boolean, String, String, String, int, i=
nt, boolean, Object[]) line: 76
MockMethodBridge.invoke(Object, Method, Object[]) line: 41
<unknown receiving type>(FrameworkMethod).invokeExplosively(Object, Objec=
t...) line: 44
InvokeMethod.evaluate() line: 17
RunBefores.evaluate() line: 26
RunAfters.evaluate() line: 27
TestWatcher$1.evaluate() line: 55
TestWatcher$1.evaluate() line: 55
TestWatcher$1.evaluate() line: 55
ExpectedException$ExpectedExceptionStatement.evaluate() line: 168
TestWatcher$1.evaluate() line: 55
RunRules.evaluate() line: 20
BlockJUnit4ClassRunner(ParentRunner<T>).runLeaf(Statement, Description, R=
unNotifier) line: 271
BlockJUnit4ClassRunner.runChild(FrameworkMethod, RunNotifier) line: 70
BlockJUnit4ClassRunner.runChild(Object, RunNotifier) line: 50
ParentRunner$3.run() line: 238
ParentRunner$1.schedule(Runnable) line: 63
BlockJUnit4ClassRunner(ParentRunner<T>).runChildren(RunNotifier) line: 23=
6
ParentRunner<T>.access$000(ParentRunner, RunNotifier) line: 53
ParentRunner$2.evaluate() line: 229
RunBefores.evaluate() line: 26
RunAfters.evaluate() line: 27
BlockJUnit4ClassRunner(ParentRunner<T>).run(RunNotifier) line: 309
JUnit4TestClassReference(JUnit4TestReference).run(TestExecution) line: 50=

TestExecution.run(ITestReference[]) line: 38
RemoteTestRunner.runTests(String[], String, TestExecution) line: 459
RemoteTestRunner.runTests(TestExecution) line: 675
RemoteTestRunner.run() line: 382
RemoteTestRunner.main(String[]) line: 192

Daniel


Chris Westin wrote:
> I seem to recall you were telling me about a new IOOB that you're seein=
g, was that you?
> Is is this?
>
> Execution Failures:
> /root/drillAutomation/framework-master/framework/resources/Functional/a=
ggregates/tpcds_variants/csv/aggregate26.q
> Query:
> select cast(case columns[0] when '' then 0 else columns[0] end as int) =
as soldd, cast(case columns[1] when '' then 0 else columns[1] end as bigi=
nt) as soldt, cast(case columns[2] when '' then 0 else columns[2] end as =
float) as itemsk, cast(case columns[3] when '' then 0 else columns[3] end=
 as decimal(18,9)) as custsk, cast(case columns[4] when '' then 0 else co=
lumns[4] end as varchar(20)) as cdemo, columns[5] as hdemo, columns[6] as=
 addrsk, columns[7] as storesk, columns[8] as promo, columns[9] as tickn,=
 sum(case columns[10] when '' then 0 else cast(columns[10] as int) end) a=
s quantities from `store_sales.dat` group by cast(case columns[0] when ''=
 then 0 else columns[0] end as int), cast(case columns[1] when '' then 0 =
else columns[1] end as bigint), cast(case columns[2] when '' then 0 else =
columns[2] end as float), cast(case columns[3] when '' then 0 else column=
s[3] end as decimal(18,9)), cast(case columns[4] when '' then 0 else colu=
mns[4] end as varchar(20)), columns[5], columns[6]
> , columns[7], columns[8], columns[9] order by soldd desc, soldt desc, i=
temsk desc limit 20
> Failed with exception
> java.sql.SQLException: SYSTEM ERROR: IndexOutOfBoundsException: index: =
0, length: 1 (expected: range(0, 0))
>
> Fragment 0:0
>
> [Error Id: 2322e296-4fff-4770-a778-c277770ea4d7 on atsqa6c61.qa.lab:310=
10]
> 	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCurso=
r.java:247)
> 	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:320)
> 	at oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.=
java:187)
> 	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetIm=
pl.java:160)
> 	at org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTes=
tJdbc.java:203)
> 	at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.jav=
a:89)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:=
471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto=
r.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut=
or.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException:=
 SYSTEM ERROR: IndexOutOfBoundsException: index: 0, length: 1 (expected: =
range(0, 0))
>
> Fragment 0:0


--=20
Daniel Barclay
MapR Technologies


--------------070006080004040604040008--