drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Pavlov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3639) Internal error: Error while applying rule PruneScanRule:Filter_On_Scan_Parquet
Date Tue, 06 Oct 2015 09:47:26 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944805#comment-14944805
] 

Ivan Pavlov commented on DRILL-3639:
------------------------------------

Sorry, but I cannot confirm this. Using the latest master, for a table partitioned by `month`
which was casted from string to int, this query:

{noformat} explain plan for select count(*) from dfs.ivan.`drill/p2008` where `month`=1; {noformat}

results in:

{noformat}
2015-10-06 02:38:49,013 [29ec6958-4361-7842-cb2e-7e1561f713a3:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Fetch parquet metadata: Executed 17 out of 17 using 16 threads. Time: 1158ms total, 898.990298ms
avg, 1156ms max.
2015-10-06 02:38:49,014 [29ec6958-4361-7842-cb2e-7e1561f713a3:foreman] INFO  o.a.d.exec.store.parquet.Metadata
- Fetch parquet metadata: Executed 17 out of 17 using 16 threads. Earliest start: 369.254000
?s, Latest start: 161749.298000 ?s, Average start: 13271.456235 ?s .
2015-10-06 02:38:49,028 [29ec6958-4361-7842-cb2e-7e1561f713a3:foreman] ERROR o.a.drill.exec.work.foreman.Foreman
- SYSTEM ERROR: StringIndexOutOfBoundsException: String index out of range: -6


[Error Id: 3bf20517-422f-4823-87cb-19e560176932 on localhost:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: StringIndexOutOfBoundsException:
String index out of range: -6


[Error Id: 3bf20517-422f-4823-87cb-19e560176932 on localhost:31010]
	at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception during
fragment initialization: Internal error: Error while applying rule PruneScanRule:Filter_On_Scan_Parquet,
args [rel#1089:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#1086:Subset#0.LOGICAL.ANY([]).[],condition==($1,
1)), rel#1102:DrillScanRel.LOGICAL.ANY([]).[](table=[dfs, ivan, drill/p2008],groupscan=ParquetGroupScan
[entries=[ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_1.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_0_2.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_3.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_4.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_0_5.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_6.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_7.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_1_1.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_1_2.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_1_3.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_1_4.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_1_5.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_1_6.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_2_1.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_2_2.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_2_3.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_2_4.parquet]], selectionRoot=hdfs://localhost:9000/ivan/drill/p2008,
numFiles=17, usedMetadataFile=false, columns=[`*`]])]
	... 4 common frames omitted
Caused by: java.lang.AssertionError: Internal error: Error while applying rule PruneScanRule:Filter_On_Scan_Parquet,
args [rel#1089:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#1086:Subset#0.LOGICAL.ANY([]).[],condition==($1,
1)), rel#1102:DrillScanRel.LOGICAL.ANY([]).[](table=[dfs, ivan, drill/p2008],groupscan=ParquetGroupScan
[entries=[ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_1.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_0_2.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_3.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_4.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_0_5.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_6.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_0_7.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_1_1.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_1_2.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_1_3.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_1_4.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_1_5.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_1_6.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_2_1.parquet], ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_2_2.parquet],
ReadEntryWithPath [path=hdfs://localhost:9000/ivan/drill/p2008/1_2_3.parquet], ReadEntryWithPath
[path=hdfs://localhost:9000/ivan/drill/p2008/1_2_4.parquet]], selectionRoot=hdfs://localhost:9000/ivan/drill/p2008,
numFiles=17, usedMetadataFile=false, columns=[`*`]])]
	at org.apache.calcite.util.Util.newInternal(Util.java:792) ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251) ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808) ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
	at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
	at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:303) ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:545)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:213)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:248)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:905) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	... 3 common frames omitted
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -6
	at java.lang.String.substring(String.java:1875) ~[na:1.7.0_79]
	at org.apache.drill.exec.planner.DFSPartitionLocation.<init>(DFSPartitionLocation.java:31)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.planner.ParquetPartitionDescriptor.createPartitionSublists(ParquetPartitionDescriptor.java:126)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.planner.AbstractPartitionDescriptor.iterator(AbstractPartitionDescriptor.java:53)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:190)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.drill.exec.planner.logical.partition.ParquetPruneScanRule$2.onMatch(ParquetPruneScanRule.java:87)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228) ~[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
	... 13 common frames omitted
{noformat}

> Internal error: Error while applying rule PruneScanRule:Filter_On_Scan_Parquet
> ------------------------------------------------------------------------------
>
>                 Key: DRILL-3639
>                 URL: https://issues.apache.org/jira/browse/DRILL-3639
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>    Affects Versions: 1.1.0
>            Reporter: 徐波
>            Assignee: Jinfeng Ni
>             Fix For: 1.2.0
>
>
> exception:
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment
initialization: Internal error: Error while applying rule PruneScanRule:Filter_On_Scan_Parquet
> Caused By (java.lang.AssertionError) null
>     org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.isBitOn():490
>     org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitBooleanAnd():434
>     org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitBooleanOperator():332
>     org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitBooleanOperator():147
>     org.apache.drill.common.expression.BooleanOperator.accept():36
> my sql:
> select col1 from hdfs.root.`/dir/table1` where col2<cast('222222222.0' as double)
and col3=101 limit 10
> col2 is not a partition column and col3 is a partition column, so the FindPartitionConditions
get a wrong condition like 'AND(CAST('222222222.0'):DOUBLE NOT NULL, =($1, 101))'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message