flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3848) Add ProjectableTableSource interface and translation rule
Date Tue, 06 Dec 2016 11:11:28 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724956#comment-15724956

ASF GitHub Bot commented on FLINK-3848:

Github user fhueske commented on the issue:

    @tonycox, great. Thanks for the notice! Is the PR ready to review from your point of view
or are you still working on it?
    @KurtYoung Thanks for reaching out to this PR. I'd like to propose the following. I'll
have a look at both PRs and see how we can maybe split the effort into two commits. So, everybody
gets a contribution out of this effort.
    Regarding making `CsvTableSource` a `ProjectableTableSource`, it is true that `CsvTableSource`
will not benefit as much as other sources. Even with pushed-down projections, it will still
read all data and split it into rows. However, we can save some time in field parsing (esp.
tailing fields are not parsed at all) and object creation (and later garbage collection).
So, making `CsvTableSource` a `ProjectableTableSource` will mainly safe some CPU time and
not IO. In addition, this is currently the only `BatchTableSource` and rather easy to test.

> Add ProjectableTableSource interface and translation rule
> ---------------------------------------------------------
>                 Key: FLINK-3848
>                 URL: https://issues.apache.org/jira/browse/FLINK-3848
>             Project: Flink
>          Issue Type: New Feature
>          Components: Table API & SQL
>            Reporter: Fabian Hueske
>            Assignee: Anton Solovev
> Add a {{ProjectableTableSource}} interface for {{TableSource}} implementation that support
projection push-down.
> The interface could look as follows
> {code}
> def trait ProjectableTableSource {
>   def setProjection(fields: Array[String]): Unit
> }
> {code}
> In addition we need Calcite rules to push a projection into a TableScan that refers to
a {{ProjectableTableSource}}. We might need to tweak the cost model as well to push the optimizer
in the right direction.
> Moreover, the {{CsvTableSource}} could be extended to implement {{ProjectableTableSource}}.

This message was sent by Atlassian JIRA

View raw message