drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacques Nadeau (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-2517) Partition pruning still reading files during planning
Date Mon, 23 Mar 2015 23:37:53 GMT

    [ https://issues.apache.org/jira/browse/DRILL-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376915#comment-14376915

Jacques Nadeau commented on DRILL-2517:

Some thoughts in no particular order:

- RelNode trees and their innards need to remain immutable.  As such, you should not be modifying
the existing selection.
- I'm not sure this is a bug. Enabling or disabling an optimizer rule like this shouldn't
change the output of the query but your request is for the rule to change the behavior.  (I'm
not absolutely set for or against this but we should discuss.)
- The behavior you are looking for can be accomplished by using glob patterns in the table
name.  For example select * from `/files/2014-05*`

> Partition pruning still reading files during planning
> -----------------------------------------------------
>                 Key: DRILL-2517
>                 URL: https://issues.apache.org/jira/browse/DRILL-2517
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>    Affects Versions: 0.7.0, 0.8.0
>            Reporter: Adam Gilmore
>            Assignee: Aman Sinha
> Partition pruning still tries to read Parquet files during the planning stage even though
they don't match the partition filter.
> For example, if there were an invalid Parquet file in a directory that should not be
> {code}
> 0: jdbc:drill:zk=local> select sum(price) from dfs.tmp.purchases where dir0 = 1;
> Query failed: IllegalArgumentException: file:/tmp/purchases/4/0_0_0.parquet is not a
Parquet file (too small)
> {code}
> The reason is that the partition pruning happens after the Parquet plugin tries to read
the footer of each file.
> Ideally, partition pruning would happen first before the format plugin gets involved.

This message was sent by Atlassian JIRA

View raw message