flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From qq <471237...@qq.com>
Subject How to understand create watermark for Kafka partitions
Date Fri, 13 Dec 2019 01:57:01 GMT
Hi all,

      I confused with watermark for each Kafka partitions.  As I know watermark  created by
data stream level. But why also say created watermark for each Kafka topic partitions ? As
I tested, watermarks also created by global, even I run my job with parallels. And assign
watermarks on Kafka consumer . Thanks .

Below text copied from flink web.

you can use Flinkā€™s Kafka-partition-aware watermark generation. Using that feature, watermarks
are generated inside the Kafka consumer, per Kafka partition, and the per-partition watermarks
are merged in the same way as watermarks are merged on stream shuffles.

For example, if event timestamps are strictly ascending per Kafka partition, generating per-partition
watermarks with the ascending timestamps watermark generator <https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/event_timestamp_extractors.html#assigners-with-ascending-timestamps>
will result in perfect overall watermarks.

The illustrations below show how to use the per-Kafka-partition watermark generation, and
how watermarks propagate through the streaming dataflow in that case.

Alex Fu
View raw message