beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From amir bahmanyari <>
Subject Avoid reading duplicates from KafkaIO!!!
Date Tue, 26 Jul 2016 20:13:25 GMT
Hi Colleagues,I run the same Beam Flink app code without reading from KafkaIO, and I get accurate
results (perfect matched records).When run exact same code but read the records from KafkaIO,
the number of matched records exceeds the accurate expected number at the end.This tells me
that I MAY BE receiving duplicates from KafkaIO causing my matched counter to increase superficially.Could
this be a possibility with KafkaIO function & how it interacts with Kafka server?If so,
how can I prevent KafkaIO from duplicate readings?ThanksAmir-
View raw message