spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Armbrust (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20002) Add support for unions between streaming and batch datasets
Date Fri, 02 Jun 2017 21:20:04 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035441#comment-16035441
] 

Michael Armbrust commented on SPARK-20002:
------------------------------------------

I'm not sure that we will ever support this.  The issue is that for batch datasets, we don't
track what has been read.  Thus its unclear what should happen when the query is restarted.
 Instead, I think you can always achieve the same result by just loading both datasets as
a stream (even if you don't plan to change one of them).  Would that work?

> Add support for unions between streaming and batch datasets
> -----------------------------------------------------------
>
>                 Key: SPARK-20002
>                 URL: https://issues.apache.org/jira/browse/SPARK-20002
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL, Structured Streaming
>    Affects Versions: 2.0.2
>            Reporter: Leon Pham
>
> Currently unions between streaming datasets and batch datasets are not supported.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message