flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Seth Wiesman <swies...@mediamath.com>
Subject Re: Watermark in broadcast
Date Wed, 13 Dec 2017 17:04:33 GMT
Hi Timo,

I think you are correct. This stream is consumed from Kafka and the number of partitions is
much less than the parallelism of the program so there would be many partitions that never
forward watermarks greater than Long.Min_Value.

Thank you for the quick response.

[cid:image001.png@01D3740A.880106E0]<http://www.mediamath.com/>

Seth Wiesman | Software Engineer, Data

4 World Trade Center, 46th Floor, New York, NY 10007




From: Timo Walther <twalthr@apache.org>
Date: Wednesday, December 13, 2017 at 11:46 AM
To: "user@flink.apache.org" <user@flink.apache.org>
Subject: Re: Watermark in broadcast

Hi Seth,

are you sure that all partitions of the broadcasted stream send a watermark? processWatermark
is only called if a minimum watermark arrived from all partitions.

Regards,
Timo

Am 12/13/17 um 5:10 PM schrieb Seth Wiesman:
Hi,

How are watermarks propagated during a broadcast partition? I have a TwoInputStreamTransformation
that takes a broadcast stream as one of its inputs. Both streams are assigned timestamps and
watermarks before being connected however I only ever see watermarks from my non-broadcast
stream. Is this expected behavior? Currently I have overridden processWatermark1 to unconditionally
call processWatermark but that does not seem like an ideal solution.

Thank you,
[cid:image001.png@01D3740A.880106E0]<http://www.mediamath.com/>

Seth Wiesman | Software Engineer, Data

4 World Trade Center, 46th Floor, New York, NY 10007





Mime
View raw message