spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jose Torres (JIRA)" <>
Subject [jira] [Commented] (SPARK-24036) Stateful operators in continuous processing
Date Thu, 10 May 2018 15:26:00 GMT


Jose Torres commented on SPARK-24036:

My concern isn't that we'll have to write more code, but that changing scheduler internals expands
the surface area of interactions that need to be considered. For example, can we confidently
enumerate all the ways in which the scheduler assumes a Dependency defines a stage boundary?
If so, can we change all of them in a way that doesn't impact non-continuous-processing code
at all? We'd have to consider a lot of questions like that, and I don't see any large benefit
we'd get from doing so.


Glad to take a look at your preview PR.

> Stateful operators in continuous processing
> -------------------------------------------
>                 Key: SPARK-24036
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Jose Torres
>            Priority: Major
> The first iteration of continuous processing in Spark 2.3 does not work with stateful

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message