drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rahul Challapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3937) We are not pruning when we have a metadata cache and auto partitioned data in some cases
Date Thu, 05 Nov 2015 18:08:28 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992143#comment-14992143
] 

Rahul Challapalli commented on DRILL-3937:
------------------------------------------

[~vicky] This looks very similar. You can try with aman's fix which is already there in master.
If that doesn't work then I would remove the metadata cache file and see if the issue goes
away so that we can isolate metadata cache file as the root cause.

> We are not pruning when we have a metadata cache and auto partitioned data in some cases
> ----------------------------------------------------------------------------------------
>
>                 Key: DRILL-3937
>                 URL: https://issues.apache.org/jira/browse/DRILL-3937
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Metadata
>            Reporter: Rahul Challapalli
>            Assignee: Aman Sinha
>             Fix For: 1.3.0
>
>         Attachments: 1_0_9998.parquet, 1_0_9999.parquet
>
>
> git.commit.id.abbrev=2736412
> The below plan indicates that we are not pruning
> {code}
> explain plan for select count(*) from dfs.`/drill/comscore/orders2` where o_clerk='Clerk#000079443';
> +------+------+
> | text | json |
> +------+------+
> | 00-00    Screen
> 00-01      Project(EXPR$0=[$0])
> 00-02        StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-03          Project($f0=[0])
> 00-04            SelectionVectorRemover
> 00-05              Filter(condition=[=($0, 'Clerk#000079443')])
> 00-06                Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///drill/comscore/orders2/1_0_9999.parquet],
ReadEntryWithPath [path=maprfs:///drill/comscore/orders2/1_0_9998.parquet]], selectionRoot=/drill/comscore/orders2,
numFiles=2, usedMetadataFile=true, columns=[`o_clerk`]]])
> {code}
> Error from the logs
> {code}
> 2015-10-15 01:24:28,467 [29e0ffb4-1c91-f40a-8bf0-5e3665dcf107:foreman] WARN  o.a.d.e.p.l.partition.PruneScanRule
- Exception while trying to prune partition.
> java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to parquet.io.api.Binary
>         at org.apache.drill.exec.store.parquet.ParquetGroupScan.populatePruningVector(ParquetGroupScan.java:414)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.ParquetPartitionDescriptor.populatePartitionVectors(ParquetPartitionDescriptor.java:96)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:212)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.logical.partition.ParquetPruneScanRule$2.onMatch(ParquetPruneScanRule.java:87)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
>         at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
>         at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) [calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
>         at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:303) [calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
>         at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:545)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:213)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:248)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:905) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> {code}
> The partition column type in this case is binary which could be causing the issue. 
> Partition pruning seems to be working when we have Metadata Caching + Auto Partitioned
Files with integer partition column 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message