drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jinfeng Ni (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.
Date Wed, 16 Dec 2015 00:21:46 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059185#comment-15059185
] 

Jinfeng Ni commented on DRILL-3765:
-----------------------------------

[~rkins], that's right. I post some preliminary performance number for "explain plan" in the
earlier comment. Please note that the improvement depends on the complexity of the partition
filtering ; essentially, this patch is trying to reduce the # of partitioning filter evaluation.
The more complex the partitioning filter is, the more likely we would see performance improvement
in planning time.    

> Partition prune rule is unnecessary fired multiple times. 
> ----------------------------------------------------------
>
>                 Key: DRILL-3765
>                 URL: https://issues.apache.org/jira/browse/DRILL-3765
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>            Reporter: Jinfeng Ni
>            Assignee: Jinfeng Ni
>             Fix For: 1.4.0
>
>
> It seems that the partition prune rule may be fired multiple times, even after the first
rule execution has pushed the filter into the scan operator. Since partition prune has to
build the vectors to contain the partition /file / directory information, to invoke the partition
prune rule unnecessary may lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to reduce
the chance of hitting OOM exception, while the partition prune rule is executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message