flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-2577) Watermarks Stall When a Source Finishes Prematurely
Date Wed, 26 Aug 2015 10:38:47 GMT
Aljoscha Krettek created FLINK-2577:

             Summary: Watermarks Stall When a Source Finishes Prematurely
                 Key: FLINK-2577
                 URL: https://issues.apache.org/jira/browse/FLINK-2577
             Project: Flink
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 0.10
            Reporter: Aljoscha Krettek
            Assignee: Aljoscha Krettek
             Fix For: 0.10

The problem with a streaming source that closes is that downstream operations never notice
that it is not running anymore and keep waiting for watermarks from all upstream operations
(including the source). This has the effect that watermarks just stop propagating through
the topology.

I think an easy fix is to change sources to emit a last watermark of +Inf before closing.
Because watermarks are always the minimum of all watermarks on the inputs this would have
the effect of advancing only depending on the other inputs.

The added benefit would be that once all sources emit a +Inf watermark the operator also get's
a last +Inf watermark which tells it that all sources are done. Right now, streaming operators
(and user code) have no way of telling if there are going to come elements in the future.
This is especially problematic in Co-Map (Co-FlatMap) operations where you have one input
that feeds a hash-table and the other input is elements that you want to stream by this hash-table.

This message was sent by Atlassian JIRA

View raw message