flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Effort to add SQL / StreamSQL to Flink
Date Thu, 07 Jan 2016 14:05:24 GMT
Hi everybody,

in the last days, Timo and I refined the design document for adding a SQL /
StreamSQL interface on top of Flink that was started by Stephan.

The document proposes an architecture that is centered around Apache
Calcite. Calcite is an Apache top-level project and includes a SQL parser,
a semantic validator for relational queries, and a rule- and cost-based
relational optimizer. Calcite is used by Apache Hive and Apache Drill
(among other projects). In a nutshell, the plan is to translate Table API
and SQL queries into Calcite's relational expression trees, optimize these
trees, and translate them into DataSet and DataStream programs.The document
breaks down the work into several tasks and subtasks.

Please review the design document and comment.

-- >
https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/edit?usp=sharing

Unless there are major concerns with the design, Timo and I want to start
next week to move the current Table API on top of Apache Calcite (Task 1 in
the document). The goal of this task is to have the same functionality as
currently, but with Calcite in the translation process. This is a blocking
task that we hope to complete soon. Afterwards, we can independently work
on different aspects such as extending the Table API, adding a SQL
interface (basically just a parser), integration with external data
sources, better code generation, optimization rules, streaming support for
the Table API, StreamSQL, etc..

Timo and I plan to work on a WIP branch to implement Task 1 and merge it to
the master branch once the task is completed. Of course, everybody is
welcome to contribute to this effort. Please let us know such that we can
coordinate our efforts.

Thanks,
Fabian

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message