drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Parth Chandra (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-1962) accessing nested array from multiple files causing IndexOutOfBoundException
Date Mon, 16 Feb 2015 22:56:12 GMT

     [ https://issues.apache.org/jira/browse/DRILL-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Parth Chandra updated DRILL-1962:
---------------------------------
    Priority: Critical  (was: Major)

> accessing nested array from multiple files causing IndexOutOfBoundException
> ---------------------------------------------------------------------------
>
>                 Key: DRILL-1962
>                 URL: https://issues.apache.org/jira/browse/DRILL-1962
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Data Types
>    Affects Versions: 0.8.0
>            Reporter: Chun Chang
>            Assignee: Parth Chandra
>            Priority: Critical
>             Fix For: 0.9.0
>
>
> #Wed Jan 07 18:54:07 EST 2015
> git.commit.id.abbrev=35a350f
> If the dataset contains nested array of array, and the data is contained in more than
one file, accessing the second nested array cause IndexOutOfBoundsException. For example,
with the following dataset:
> {code}
> {
>     "id": 2,
>     "oooa": {
>         "oa": {
>             "oab": {
>                 "oabc": [
>                     {
>                         "rowId": 2
>                     },
>                     {
>                         "rowValue1": [{"rv1":1, "rv2":2}, {"rva1":3, "rva2":4}],
>                         "rowValue2": [{"rw1":1, "rw2":2}, {"rwa1":3, "rwa2":4}]
>                     }
>                 ]
>             }
>         }
>     }
> }
> {code}
> If you put it in two separate files in the same directory, query using wild card to accessing
the two files at the second array level will cause the exception.
> {code}
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select t.oooa.oa.oab.oabc[1].rowValue1
from `jira2file/jira*.json` t;
> +------------+
> |   EXPR$0   |
> +------------+
> | [{"rv1":1,"rv2":2},{"rva1":3,"rva2":4}] |
> Query failed: Query failed: Failure while running fragment., index: -4, length: 4 (expected:
range(0, 16384)) [ 78235243-4f01-4ee3-9675-fc18bd1e66e3 on qa-node120.qa.lab:31010 ]
> [ 78235243-4f01-4ee3-9675-fc18bd1e66e3 on qa-node120.qa.lab:31010 ]
> java.lang.RuntimeException: java.sql.SQLException: Failure while executing query.
> 	at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
> 	at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
> 	at sqlline.SqlLine.print(SqlLine.java:1809)
> 	at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
> 	at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
> 	at sqlline.SqlLine.dispatch(SqlLine.java:889)
> 	at sqlline.SqlLine.begin(SqlLine.java:763)
> 	at sqlline.SqlLine.start(SqlLine.java:498)
> 	at sqlline.SqlLine.main(SqlLine.java:460)
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ>
> {code}
> Same query on single file works.
> {code}
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select t.oooa.oa.oab.oabc[1].rowValue1
from `jira2file/jira1.json` t;
> +------------+
> |   EXPR$0   |
> +------------+
> | [{"rv1":1,"rv2":2},{"rva1":3,"rva2":4}] |
> +------------+
> {code}
> stack trace:
> {code}
> 2015-01-08 14:00:16,127 [2b51020e-daab-a903-ef7c-ef6f9bd606c7:frag:0:0] WARN  o.a.d.e.w.fragment.FragmentExecutor
- Error while initializing or executing fragment
> java.lang.IndexOutOfBoundsException: index: -4, length: 4 (expected: range(0, 16384))
> 	at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:156) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
> 	at io.netty.buffer.DrillBuf.chk(DrillBuf.java:178) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
> 	at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:447) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
> 	at org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:297) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapAccessor.get(RepeatedMapVector.java:542)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:90)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setChildrenPosition(RepeatedMapReaderImpl.java:45)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:96)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.test.generated.ProjectorGen4732.doEval(ProjectorTemplate.java:30)
~[na:na]
> 	at org.apache.drill.exec.test.generated.ProjectorGen4732.projectRecords(ProjectorTemplate.java:64)
~[na:na]
> 	at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:172)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:97)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:114)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:254)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
> 	at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> {code}
> physical plan:
> {code}
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> explain plan for select t.oooa.oa.oab.oabc[1].rowValue1
from `jira2file/jira*.json` t;
> +------------+------------+
> |    text    |    json    |
> +------------+------------+
> | 00-00    Screen
> 00-01      Project(EXPR$0=[ITEM(ITEM(ITEM(ITEM(ITEM($0, 'oa'), 'oab'), 'oabc'), 1), 'rowValue1')])
> 00-02        Scan(groupscan=[EasyGroupScan [selectionRoot=/drill/testdata/complex_type/json/jira2file,
numFiles=2, columns=[`oooa`.`oa`.`oab`.`oabc`[1].`rowValue1`], files=[maprfs:/drill/testdata/complex_type/json/jira2file/jira1.json,
maprfs:/drill/testdata/complex_type/json/jira2file/jira2.json]]])
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message