drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Altekruse (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-2173) Enable querying partition information without reading all data
Date Thu, 05 Feb 2015 22:47:36 GMT
Jason Altekruse created DRILL-2173:

             Summary: Enable querying partition information without reading all data
                 Key: DRILL-2173
                 URL: https://issues.apache.org/jira/browse/DRILL-2173
             Project: Apache Drill
          Issue Type: New Feature
          Components: Query Planning & Optimization
    Affects Versions: 0.7.0
            Reporter: Jason Altekruse
            Assignee: Jason Altekruse

When reading a series of files in nested directories, Drill currently adds columns representing
the directory structure that was traversed to reach the file currently being read. These columns
are stored as varchar under tha names dir0, dir1, ...  As these are just regular columns,
Drill allows arbitrary queries against this data, in terms of aggregates, filter, sort, etc.
To allow optimizing reads, basic partition pruning has already been added to prune in the
case of an expression like dir0 = "2015" or a simple in list, which is converted during planning
to a series of ORs of equals expressions. If users want to query the directory information
dynamically, and not include specific directory names in the query, this will prompt a full
table scan and filter operation on the dir columns. This enhancement is to allow more complex
queries to be run against directory metadata, and only scanning the matching directories.

This message was sent by Atlassian JIRA

View raw message