Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Fri, 4 Nov 2016 12:28:58 +0000 (UTC)
From: "Tzu-Li (Gordon) Tai (JIRA)" <jira@apache.org>
To: dev@flink.apache.org
Message-ID: <JIRA.13018054.1478262493000.188117.1478262538439@Atlassian.JIRA>
In-Reply-To: <JIRA.13018054.1478262493000@Atlassian.JIRA>
References: <JIRA.13018054.1478262493000@Atlassian.JIRA> <JIRA.13018054.1478262493640@arcas>
Subject: [jira] [Created] (FLINK-5017) Introduce WatermarkStatus stream
 element to allow for temporarily idle streaming sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 04 Nov 2016 12:29:00 -0000

Tzu-Li (Gordon) Tai created FLINK-5017:
------------------------------------------

             Summary: Introduce WatermarkStatus stream element to allow for=
 temporarily idle streaming sources
                 Key: FLINK-5017
                 URL: https://issues.apache.org/jira/browse/FLINK-5017
             Project: Flink
          Issue Type: New Feature
          Components: Streaming
            Reporter: Tzu-Li (Gordon) Tai
            Assignee: Tzu-Li (Gordon) Tai
            Priority: Blocker
             Fix For: 1.2.0


A {{WatermarkStatus}} element informs receiving operators whether or not th=
ey should continue to expect watermarks from the sending operator. There ar=
e 2 kinds of status, namely {{IDLE}} and {{ACTIVE}}. Watermark status eleme=
nts are generated at the sources, and may be propagated through the operato=
rs of the topology using {{Output#emitWatermarkStatus(WatermarkStatus)}}.
Sources and downstream operators should emit either of the status elements =
once it changes between "watermark-idle" and "watermark-active" states.

A source is considered "watermark-idle" if it will not emit records for an =
indefinite amount of time. This is the case, for example, for Flink's Kafka=
 Consumer, where sources might initially have no assigned partitions to rea=
d from, or no records can be read from the assigned partitions. Once the so=
urce detects that it will resume emitting data, it is considered "watermark=
-active".

Downstream operators with multiple inputs (ex. head operators of a {{OneInp=
utStreamTask}} or {{TwoInputStreamTask}}) should not wait for watermarks fr=
om an upstream operator that is "watermark-idle" when deciding whether or n=
ot to advance the operator's current watermark. When a downstream operator =
determines that all upstream operators are "watermark-idle" (i.e. when all =
input channels have received the watermark idle status element), then the o=
perator is considered to also be "watermark-idle", as it will temporarily b=
e unable to advance its own watermark. This is always the case for operator=
s that only read from a single upstream operator. Once an operator is consi=
dered "watermark-idle", it should itself forward its idle status to inform =
downstream operators. The operator is considered to be back to "watermark-a=
ctive" as soon as at least one of its upstream operators resume to be "wate=
rmark-active" (i.e. when at least one input channel receives the watermark =
active status element), and should also forward its active status to inform=
 downstream operators.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)