drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aman Sinha (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-3759) Make partition pruning multi-phased to reduce the working set kept in memory
Date Thu, 10 Sep 2015 19:01:45 GMT
Aman Sinha created DRILL-3759:
---------------------------------

             Summary: Make partition pruning multi-phased to reduce the working set kept in
memory
                 Key: DRILL-3759
                 URL: https://issues.apache.org/jira/browse/DRILL-3759
             Project: Apache Drill
          Issue Type: Improvement
          Components: Query Planning & Optimization
    Affects Versions: 1.1.0
            Reporter: Aman Sinha
            Assignee: Jinfeng Ni


Currently, partition pruning gets all file names in the table and applies the pruning.  Suppose
the files are spread out over several directories and there is a filter  on dirN,  this is
not efficient - both in terms of elapsed time and memory usage.  This has been seen in a few
use cases recently. 

Wherever possible, we should ideally perform the pruning in N steps (where N is the number
of directory levels referenced in the filter conditions):   
  1. Get the directory and  filenames at level i
  2. Materialize into the in-memory table 
  3. Apply interpreter-based evaluation of filter condition
  4. Determine qualifying directories, increment i and repeat from step 1
 
This multi phase approach may not be possible for certain types of filters - e,g for disjunctions.
This analysis needs to be done. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message