flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: getting duplicate messages from duplicate jobs
Date Tue, 29 Jan 2019 07:46:58 GMT
Hi,

Yes, Dawid is correct.

The "group.id" setting in Flink's Kafka Consumer is only used for group
offset fetching and committing offsets back to Kafka (only for exposure
purposes, not used for processing guarantees).
The Flink Kafka Consumer uses static partition assignment on the
KafkaConsumer API, and not consumer group-based automatic partition
assignments.

Cheers,
Gordon

On Sun, Jan 27, 2019 at 12:28 AM Dawid Wysakowicz <dwysakowicz@apache.org>
wrote:

> Forgot to cc Gordon :)
>
> On 23/01/2019 18:02, Avi Levi wrote:
> > Hi,
> > This quite confusing.
> > I submitted the same stateless job twice (actually I upload it once).
> > However when I place a message on kafka, it seems that both jobs
> > consumes it, and publish the same result (we publish the result to
> > other kafka topic, so I actually see the massage duplicated on kafka
> > ). how can it be ? both jobs are using the same group id (group id is
> > fixed and not generated )
> >
> > Kind regards
> > Avi
>
>

Mime
View raw message