drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khurram Faraaz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-2912) Exception is not propagated correctly in case when directory contains mix of file types
Date Wed, 13 Jan 2016 10:11:39 GMT

    [ https://issues.apache.org/jira/browse/DRILL-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095934#comment-15095934
] 

Khurram Faraaz commented on DRILL-2912:
---------------------------------------

There is a related assertion that is seen when assertions are enabled on MapR Drill 1.4.0
git.commit.id=556d9c5d
Can someone take a look into this ?

{noformat}
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception during
fragment initialization: Internal error: Error while applying rule DrillPushProjIntoScan,
args [rel#803367:LogicalProject.NONE.ANY([]).[](input=rel#803366:Subset#0.ENUMERABLE.ANY([]).[],$f0=0),
rel#803358:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, /drill/testdata/metadata_caching/lineitem_deletecache/])]
        ... 4 common frames omitted
Caused by: java.lang.AssertionError: Internal error: Error while applying rule DrillPushProjIntoScan,
args [rel#803367:LogicalProject.NONE.ANY([]).[](input=rel#803366:Subset#0.ENUMERABLE.ANY([]).[],$f0=0),
rel#803358:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, /drill/testdata/metadata_caching/lineitem_deletecache/])]
        at org.apache.calcite.util.Util.newInternal(Util.java:792) ~[calcite-core-1.4.0-drill-1.4.0-mapr-r1.jar:1.4.0-drill-1.4.0-mapr-r1]
        at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
~[calcite-core-1.4.0-drill-1.4.0-mapr-r1.jar:1.4.0-drill-1.4.0-mapr-r1]
        at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
~[calcite-core-1.4.0-drill-1.4.0-mapr-r1.jar:1.4.0-drill-1.4.0-mapr-r1]
        at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) ~[calcite-core-1.4.0-drill-1.4.0-mapr-r1.jar:1.4.0-drill-1.4.0-mapr-r1]
        at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:313) ~[calcite-core-1.4.0-drill-1.4.0-mapr-r1.jar:1.4.0-drill-1.4.0-mapr-r1]
        at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.doLogicalPlanning(DefaultSqlHandler.java:562)
~[drill-java-exec-1.4.0.jar:1.4.0]
        at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:230)
~[drill-java-exec-1.4.0.jar:1.4.0]
        at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:263)
~[drill-java-exec-1.4.0.jar:1.4.0]
        at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:172)
~[drill-java-exec-1.4.0.jar:1.4.0]
        at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:218)
~[drill-java-exec-1.4.0.jar:1.4.0]
        at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:909) [drill-java-exec-1.4.0.jar:1.4.0]
        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244) [drill-java-exec-1.4.0.jar:1.4.0]
        ... 3 common frames omitted
{noformat}

> Exception is not propagated correctly in case when directory contains mix of file types
> ---------------------------------------------------------------------------------------
>
>                 Key: DRILL-2912
>                 URL: https://issues.apache.org/jira/browse/DRILL-2912
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Flow, Storage - JSON, Storage - Parquet
>            Reporter: Victoria Markman
>             Fix For: Future
>
>
> While trying to read from directory that has a mix of parquet and json files I ran into
an exception:
> {code}
> 0: jdbc:drill:schema=dfs> select max(dir0) from bigtable;
> Query failed: SYSTEM ERROR: Unexpected exception during fragment initialization: Internal
error: Error while applying rule DrillPushProjIntoScan, args [rel#4207:LogicalProject.NONE.ANY([]).[](input=rel#4206:Subset#0.ENUMERABLE.ANY([]).[],dir0=$1),
rel#4198:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, test, bigtable])]
> [72d7f7ee-3045-44d9-b13c-1d03bea4e22c on atsqa4-133.qa.lab:31010]
> Error: exception while executing query: Failure while executing query. (state=,code=0)
> {code}
> The real problem is that directory contains 2 parquet and one json files:
> {code}
> [Wed Apr 29 14:50:58 root@/mapr/vmarkman.cluster.com/test/bigtable/F114/2014-03-27 ]
# pwd
> /mapr/vmarkman.cluster.com/test/bigtable/F114/2014-03-27
> [Wed Apr 29 14:51:06 root@/mapr/vmarkman.cluster.com/test/bigtable/F114/2014-03-27 ]
# ls -ltr
> total 2
> -rwxr-xr-x 1 root root 483 Apr 16 16:05 0_0_0.parquet
> -rwxr-xr-x 1 root root 483 Apr 17 13:06 214c279334946e65-7e32c56eed93cbc2_1965630551_data.0.parq
> -rw-r--r-- 1 root root  17 Apr 23 15:24 t1.json
> {code}
> drillbit.log
> {code}
> [72d7f7ee-3045-44d9-b13c-1d03bea4e22c on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: Unexpected exception
during fragment initialization: Internal error: Error while applying rule DrillPushProjIntoScan,
args [rel#4207:LogicalProject.NONE.ANY([]).[](input=rel#4206:Subset#0.ENUMERABLE.ANY([]).[],dir0=$1),
rel#4198:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, test, bigtable])]
> [72d7f7ee-3045-44d9-b13c-1d03bea4e22c on atsqa4-133.qa.lab:31010]
>         at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:465)
~[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:620)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:717)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:659)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) [drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:661)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:762) [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:212) [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
during fragment initialization: Internal error: Error while applying rule DrillPushProjIntoScan,
args [rel#4207:LogicalProject.NONE.ANY([]).[](input=rel#4206:Subset#0.ENUMERABLE.ANY([]).[],dir0=$1),
rel#4198:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, test, bigtable])]
>         ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying rule DrillPushProjIntoScan,
args [rel#4207:LogicalProject.NONE.ANY([]).[](input=rel#4206:Subset#0.ENUMERABLE.ANY([]).[],dir0=$1),
rel#4198:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, test, bigtable])]
>         at org.apache.calcite.util.Util.newInternal(Util.java:743) ~[calcite-core-1.1.0-drill-r2.jar:1.1.0-drill-r2]
>         at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
~[calcite-core-1.1.0-drill-r2.jar:1.1.0-drill-r2]
>         at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
~[calcite-core-1.1.0-drill-r2.jar:1.1.0-drill-r2]
>         at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) ~[calcite-core-1.1.0-drill-r2.jar:1.1.0-drill-r2]
>         at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:316) ~[calcite-core-1.1.0-drill-r2.jar:1.1.0-drill-r2]
>         at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:241)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:152)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:167)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773) [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:203) [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         ... 3 common frames omitted
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Failed to read footer
entries from parquet input files: maprfs:/test/bigtable/F114/2014-03-27/t1.json is not a Parquet
file. expected magic number at tail [80, 65, 82, 49] but found [34, 32, 125, 10]
>         at org.apache.drill.exec.store.parquet.ParquetGroupScan.readFooter(ParquetGroupScan.java:213)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.parquet.ParquetGroupScan.<init>(ParquetGroupScan.java:153)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:166)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:64)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:126)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:53)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:72)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch(DrillPushProjIntoScan.java:53)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
~[calcite-core-1.1.0-drill-r2.jar:1.1.0-drill-r2]
>         ... 11 common frames omitted
> Caused by: java.io.IOException: maprfs:/test/bigtable/F114/2014-03-27/t1.json is not
a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [34, 32, 125, 10]
>         at org.apache.drill.exec.store.parquet.FooterGatherer.checkMagicBytes(FooterGatherer.java:63)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.parquet.FooterGatherer.readFooter(FooterGatherer.java:144)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.runInner(FooterGatherer.java:115)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.runInner(FooterGatherer.java:102)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:47) ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at org.apache.drill.exec.store.TimedRunnable$LatchedRunnable.run(TimedRunnable.java:86)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_71]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_71]
>         ... 3 common frames omitted
> {code}
> This is a real problem:
> {code}
> Caused by: java.io.IOException: maprfs:/test/bigtable/F114/2014-03-27/t1.json is not
a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [34, 32, 125, 10]
>         at 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message