flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avi Levi <avi.l...@bluevoyant.com>
Subject Re: getting duplicate messages from duplicate jobs
Date Wed, 30 Jan 2019 12:41:06 GMT
Ok, if you guys think it's should be like that then so be it. All I am
saying is that it is not standard behaviour from kafka consumer, at least
according to the documentation
<https://kafka.apache.org/documentation/#intro_consumers> . I understand
that flink implements things differently and all I am saying that this is
not according to kafka standard consumer group.


On Tue, Jan 29, 2019 at 9:47 AM Tzu-Li (Gordon) Tai <tzulitai@apache.org>
wrote:

> Hi,
>
> Yes, Dawid is correct.
>
> The "
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__group.id&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=dpWtkT5FJRWFqDA3MAnB4-dRYGDQjgfQTYAocqGkRKo&m=KOx07VP_axxlgt0eYEnVHNux-bJCoSwzMInyMaKqCR4&s=6pT15T_bD5q0Rbzla6yl5aqek0zSwEhMI41vxALQAKg&e=>
> group.id
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__group.id&d=DwQFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=dpWtkT5FJRWFqDA3MAnB4-dRYGDQjgfQTYAocqGkRKo&m=KOx07VP_axxlgt0eYEnVHNux-bJCoSwzMInyMaKqCR4&s=6pT15T_bD5q0Rbzla6yl5aqek0zSwEhMI41vxALQAKg&e=>"
> setting in Flink's Kafka Consumer is only used for group offset fetching
> and committing offsets back to Kafka (only for exposure purposes, not used
> for processing guarantees).
> The Flink Kafka Consumer uses static partition assignment on the
> KafkaConsumer API, and not consumer group-based automatic partition
> assignments.
>
> Cheers,
> Gordon
>
> On Sun, Jan 27, 2019 at 12:28 AM Dawid Wysakowicz <dwysakowicz@apache.org>
> wrote:
>
>> Forgot to cc Gordon :)
>>
>> On 23/01/2019 18:02, Avi Levi wrote:
>> > Hi,
>> > This quite confusing.
>> > I submitted the same stateless job twice (actually I upload it once).
>> > However when I place a message on kafka, it seems that both jobs
>> > consumes it, and publish the same result (we publish the result to
>> > other kafka topic, so I actually see the massage duplicated on kafka
>> > ). how can it be ? both jobs are using the same group id (group id is
>> > fixed and not generated )
>> >
>> > Kind regards
>> > Avi
>>
>>

Mime
View raw message