spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jungtaek Lim (JIRA)" <>
Subject [jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark
Date Mon, 08 Oct 2018 03:54:00 GMT


Jungtaek Lim commented on SPARK-24630:

[~Jackey Lee]

For DDL it would be better to participate discussion on Spark dev. mailing list which Ryan
Blue is driving.

For 'stream' keyword, I'm not sure how using the keyword 'stream' makes huge difference. It
can be a kind of marker to verify at least one of sources is stream as well as sink is stream,
but while I guess this proposal is inspired by Streaming SQL in Flink or Calcite, even Flink
they no longer using keyword 'stream'. "STREAM" is not available in BNF-grammar in here: []

Spark is on the same page, even without stream keyword, once Spark determines whether one
of source(s) and sink are both stream, query will work as structured streaming.


> SPIP: Support SQLStreaming in Spark
> -----------------------------------
>                 Key: SPARK-24630
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.2.0, 2.2.1
>            Reporter: Jackey Lee
>            Priority: Minor
>              Labels: SQLStreaming
>         Attachments: SQLStreaming SPIP.pdf
> At present, KafkaSQL, Flink SQL(which is actually based on Calcite), SQLStream, StormSQL
all provide a stream type SQL interface, with which users with little knowledge about streaming, 
can easily develop a flow system processing model. In Spark, we can also support SQL API based
on StructStreamig.
> To support for SQL Streaming, there are two key points: 
> 1, Analysis should be able to parse streaming type SQL. 
> 2, Analyzer should be able to map metadata information to the corresponding 
> Relation. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message