flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ljwagerfield <lawre...@dmz.wagerfield.com>
Subject Deduplicate messages from Kafka topic
Date Sat, 14 Jan 2017 17:11:34 GMT
As I understand it, the Flink Kafka Producer may emit duplicates to Kafka

How can I deduplicate these messages when reading them back with Flink (via
the Flink Kafka Consumer)?

For example, is there any out-the-box support for deduplicating messages,
i.e. by incorporating something like "idempotent producers" as proposed by
Jay Krepps (which, as I understand it, involves maintaining a "high
watermark" on a message-by-message level)?

View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Deduplicate-messages-from-Kafka-topic-tp11051.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

View raw message