spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Commented] (SPARK-23503) continuous execution should sequence committed epochs
Date Thu, 29 Mar 2018 14:48:00 GMT


Apache Spark commented on SPARK-23503:

User 'efimpoberezkin' has created a pull request for this issue:

> continuous execution should sequence committed epochs
> -----------------------------------------------------
>                 Key: SPARK-23503
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Jose Torres
>            Priority: Major
> Currently, the EpochCoordinator doesn't enforce a commit order. If a message for epoch
n gets lost in the ether, and epoch n + 1 happens to be ready for commit earlier, epoch n
+ 1 will be committed.
> This is either incorrect or needlessly confusing, because it's not safe to start from
the end offset of epoch n + 1 until epoch n is committed. EpochCoordinator should enforce
this sequencing.
> Note that this is not actually a problem right now, because the commit messages go through
the same RPC channel from the same place. But we shouldn't implicitly bake this assumption

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message