flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fabian Hueske (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5859) support partition pruning on Table API & SQL
Date Thu, 23 Feb 2017 13:29:44 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880416#comment-15880416

Fabian Hueske commented on FLINK-5859:

For such cases, we could either 

1. implement {{FilterableTableSource}} and manually figure out filters and partitions or 
2. {{PartitionableTableSource}} could have another method {{setFilterPredicate()}} which has
the same semantics as {{FilterableTableSource.setPredicate()}} but which is called from {{PartitionableTableSource.setPredicate()}}
with the remaining predicates which could not be used to prune partitions.

> support partition pruning on Table API & SQL
> --------------------------------------------
>                 Key: FLINK-5859
>                 URL: https://issues.apache.org/jira/browse/FLINK-5859
>             Project: Flink
>          Issue Type: New Feature
>          Components: Table API & SQL
>            Reporter: godfrey he
>            Assignee: godfrey he
> Many data sources are partitionable storage, e.g. HDFS, Druid. And many queries just
need to read a small subset of the total data. We can use partition information to prune or
skip over files irrelevant to the user’s queries. Both query optimization time and execution
time can be reduced obviously, especially for a large partitioned table.

This message was sent by Atlassian JIRA

View raw message