drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arina Ielchiieva (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-6259) Implement parquet filter push down for complex types
Date Fri, 16 Mar 2018 15:21:00 GMT
Arina Ielchiieva created DRILL-6259:

             Summary: Implement parquet filter push down for complex types
                 Key: DRILL-6259
                 URL: https://issues.apache.org/jira/browse/DRILL-6259
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.13.0
            Reporter: Arina Ielchiieva
            Assignee: Arina Ielchiieva
             Fix For: 1.14.0

Currently parquet filter push down is not working for complex types (including arrays).

This Jira aims to implement filter push down for complex types which underneath type is among
supported simple types for filter push down. For instance, currently Drill does not support
filter push down for varchars, decimals etc. Though once Drill will start support, this support
will be applied for complex type automatically.

Complex fields will be pushed down the same way regular fields are, except for one case with

Query with predicate {{where users.hobbies_ids[2] is null}} won't be able to push down because
we are not able to determine exact number of nulls in arrays fields. 

{{Consider [1, 2, 3]}} vs {{[1, 2]. If}} these arrays are in different files. Statistics
for the second case won't show any nulls but when querying from two files, in terms of data
the third value in array is null.


This message was sent by Atlassian JIRA

View raw message