drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3937) We are not pruning when we have a metadata cache and auto partitioned data in some cases
Date Thu, 29 Oct 2015 21:05:27 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981271#comment-14981271
] 

ASF GitHub Bot commented on DRILL-3937:
---------------------------------------

Github user mehant commented on a diff in the pull request:

    https://github.com/apache/drill/pull/220#discussion_r43444174
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/Metadata.java
---
    @@ -494,5 +497,48 @@ public ColumnMetadata(SchemaPath name, PrimitiveTypeName primitiveType,
Original
           this.min = min;
           this.nulls = nulls;
         }
    +
    +    @JsonProperty(value = "min")
    +    public Object getMin() {
    +      if (primitiveType == PrimitiveTypeName.BINARY) {
    +        if (originalType == OriginalType.UTF8) {
    --- End diff --
    
    Had an offline discussion with Aman, we can remove this check and serialize everything
using getBytes()


> We are not pruning when we have a metadata cache and auto partitioned data in some cases
> ----------------------------------------------------------------------------------------
>
>                 Key: DRILL-3937
>                 URL: https://issues.apache.org/jira/browse/DRILL-3937
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Metadata
>            Reporter: Rahul Challapalli
>            Assignee: Aman Sinha
>         Attachments: 1_0_9998.parquet, 1_0_9999.parquet
>
>
> git.commit.id.abbrev=2736412
> The below plan indicates that we are not pruning
> {code}
> explain plan for select count(*) from dfs.`/drill/comscore/orders2` where o_clerk='Clerk#000079443';
> +------+------+
> | text | json |
> +------+------+
> | 00-00    Screen
> 00-01      Project(EXPR$0=[$0])
> 00-02        StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-03          Project($f0=[0])
> 00-04            SelectionVectorRemover
> 00-05              Filter(condition=[=($0, 'Clerk#000079443')])
> 00-06                Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=maprfs:///drill/comscore/orders2/1_0_9999.parquet],
ReadEntryWithPath [path=maprfs:///drill/comscore/orders2/1_0_9998.parquet]], selectionRoot=/drill/comscore/orders2,
numFiles=2, usedMetadataFile=true, columns=[`o_clerk`]]])
> {code}
> Error from the logs
> {code}
> 2015-10-15 01:24:28,467 [29e0ffb4-1c91-f40a-8bf0-5e3665dcf107:foreman] WARN  o.a.d.e.p.l.partition.PruneScanRule
- Exception while trying to prune partition.
> java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to parquet.io.api.Binary
>         at org.apache.drill.exec.store.parquet.ParquetGroupScan.populatePruningVector(ParquetGroupScan.java:414)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.ParquetPartitionDescriptor.populatePartitionVectors(ParquetPartitionDescriptor.java:96)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:212)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.logical.partition.ParquetPruneScanRule$2.onMatch(ParquetPruneScanRule.java:87)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
>         at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
>         at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) [calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
>         at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:303) [calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
>         at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:545)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:213)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:248)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:905) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> {code}
> The partition column type in this case is binary which could be causing the issue. 
> Partition pruning seems to be working when we have Metadata Caching + Auto Partitioned
Files with integer partition column 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message