Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Wed, 7 Oct 2015 12:45:26 +0000 (UTC)
From: "Timo Walther (JIRA)" <jira@apache.org>
To: dev@flink.apache.org
Message-ID: <JIRA.12902973.1444221893000.45678.1444221926499@Atlassian.JIRA>
In-Reply-To: <JIRA.12902973.1444221893000@Atlassian.JIRA>
References: <JIRA.12902973.1444221893000@Atlassian.JIRA>
 <JIRA.12902973.1444221893346@arcas>
Subject: [jira] [Created] (FLINK-2828) Add interfaces for Table API input
 formats
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Timo Walther created FLINK-2828:
-----------------------------------

             Summary: Add interfaces for Table API input formats
                 Key: FLINK-2828
                 URL: https://issues.apache.org/jira/browse/FLINK-2828
             Project: Flink
          Issue Type: New Feature
          Components: Table API
            Reporter: Timo Walther
            Assignee: Timo Walther


In order to support input formats for the Table API, interfaces are necessary. I propose two types of TableSources:

- AdaptiveTableSources can adapt their output to the requirements of the plan. Although the output schema stays the same, the TableSource can react on field resolution and/or predicates internally and can return adapted DataSet/DataStream versions in the "translate" step.
- StaticTableSources are an easy way to provide the Table API with additional input formats without much implementation effort (e.g. for fromCsvFile())

TableSources need to be deeply integrated into the Table API.

The TableEnvironment requires a newly introduced AbstractExecutionEnvironment (common super class of all ExecutionEnvironments for DataSets and DataStreams).

Here's what a TableSource can see from more complicated queries:

{code}
getTableJava(tableSource1)
  .filter("a===5 || a===6")
  .select("a as a4, b as b4, c as c4")
  .filter("b4===7")
  .join(getTableJava(tableSource2))
  .where("a===a4 && c==='Test' && c4==='Test2'")

// Result predicates for tableSource1:
//  List("a===5 || a===6", "b===7", "c==='Test2'")
// Result predicates for tableSource2:
//  List("c==='Test'")
// Result resolved fields for tableSource1 (true = filtering, false=selection):
//  Set(("a", true), ("a", false), ("b", true), ("b", false), ("c", false), ("c", true))
// Result resolved fields for tableSource2 (true = filtering, false=selection):
//  Set(("a", true), ("c", true))
{code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)