drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.
Date Mon, 16 Nov 2015 18:50:10 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007101#comment-15007101
] 

ASF GitHub Bot commented on DRILL-3765:
---------------------------------------

Github user amansinha100 commented on a diff in the pull request:

    https://github.com/apache/drill/pull/255#discussion_r44964325
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
---
    @@ -116,20 +117,32 @@ public DrillSqlWorker(QueryContext context) {
     
       private RuleSet[] getRules(QueryContext context) {
         StoragePluginRegistry storagePluginRegistry = context.getStorage();
    -    RuleSet drillLogicalRules = DrillRuleSets.mergedRuleSets(
    +
    +    // Ruleset for the case where VolcanoPlanner is used for everything : join, filter/project
pushdown, partition pruning.
    +    RuleSet drillLogicalVolOnlyRules = DrillRuleSets.mergedRuleSets(
             DrillRuleSets.getDrillBasicRules(context),
    +        DrillRuleSets.getPruneScanRules(context),
             DrillRuleSets.getJoinPermRules(context),
             DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
    +
    +    // Ruleset for the case where join planning is done in Hep-LOPT, filter/project pushdown
and parttion pruning are done in VolcanoPlanner
    +    RuleSet drillLogicalHepJoinRules = DrillRuleSets.mergedRuleSets(
    +        DrillRuleSets.getDrillBasicRules(context),
    +        DrillRuleSets.getPruneScanRules(context),
    +        DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
    +
    +    // Ruleset for the case where join planning and partition pruning is done in Hep,
filter/project pushdown are done in VolcanoPlanner
    --- End diff --
    
    This comment refers to partition pruning rules but the ruleset does not have those..


> Partition prune rule is unnecessary fired multiple times. 
> ----------------------------------------------------------
>
>                 Key: DRILL-3765
>                 URL: https://issues.apache.org/jira/browse/DRILL-3765
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization
>            Reporter: Jinfeng Ni
>            Assignee: Aman Sinha
>
> It seems that the partition prune rule may be fired multiple times, even after the first
rule execution has pushed the filter into the scan operator. Since partition prune has to
build the vectors to contain the partition /file / directory information, to invoke the partition
prune rule unnecessary may lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to reduce
the chance of hitting OOM exception, while the partition prune rule is executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message