flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: flink-kafka-connector offset management
Date Fri, 20 May 2016 10:09:58 GMT
Hey Arun!

How did you configure your Kafka source? If the offset has been
committed and you configured the source to read from the latest
offset, the message should not be re-processed.

– Ufuk

On Fri, May 13, 2016 at 2:19 PM, Arun Balan <arunbalan@gmail.com> wrote:
> Hi, I am trying to use the flink-kafka-connector and I notice that every
> time I restart my application it re-reads the last message on the kafka
> topic. So if the latest offset on the topic is 10, then when the application
> is restarted, kafka-connector will re-read message 10. Why is this the
> behavior? I would assume that the last message has already been read and
> offset committed. I require that messages that are already processed from
> the topic not be reprocessed. Any insight would be helpful.
> Thanks
> Arun Balan

View raw message