spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cristian Opris (JIRA)" <>
Subject [jira] [Created] (SPARK-18721) ForeachSink breaks Watermark in append mode
Date Mon, 05 Dec 2016 21:10:58 GMT
Cristian Opris created SPARK-18721:

             Summary: ForeachSink breaks Watermark in append mode
                 Key: SPARK-18721
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.1.0
            Reporter: Cristian Opris
            Priority: Critical

The watermark is not updated in append mode with a ForeachSink

Because ForeachSink creates a separate IncrementalExecution instance, the physical plan will
be recreated for the logical plan, which results in a new EventTimeWatermarkExec operator
being created, that's unreachable from StreamExecution. This results in the watermark never
being updated, and append mode never emits results.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message