flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Selvaraj chennappan <selvarajchennap...@gmail.com>
Subject Re: getting duplicate messages from duplicate jobs
Date Wed, 30 Jan 2019 13:46:54 GMT
I have faced same problem .
https://stackoverflow.com/questions/54286486/two-kafka-consumer-in-same-group-and-one-partition


On Wed, Jan 30, 2019 at 6:11 PM Avi Levi <avi.levi@bluevoyant.com> wrote:

> Ok, if you guys think it's should be like that then so be it. All I am
> saying is that it is not standard behaviour from kafka consumer, at least
> according to the documentation
> <https://kafka.apache.org/documentation/#intro_consumers> . I understand
> that flink implements things differently and all I am saying that this is
> not according to kafka standard consumer group.
>
>
> On Tue, Jan 29, 2019 at 9:47 AM Tzu-Li (Gordon) Tai <tzulitai@apache.org>
> wrote:
>
>> Hi,
>>
>> Yes, Dawid is correct.
>>
>> The "
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__group.id&d=DwMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=dpWtkT5FJRWFqDA3MAnB4-dRYGDQjgfQTYAocqGkRKo&m=KOx07VP_axxlgt0eYEnVHNux-bJCoSwzMInyMaKqCR4&s=6pT15T_bD5q0Rbzla6yl5aqek0zSwEhMI41vxALQAKg&e=>
>> group.id
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__group.id&d=DwQFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=dpWtkT5FJRWFqDA3MAnB4-dRYGDQjgfQTYAocqGkRKo&m=KOx07VP_axxlgt0eYEnVHNux-bJCoSwzMInyMaKqCR4&s=6pT15T_bD5q0Rbzla6yl5aqek0zSwEhMI41vxALQAKg&e=>"
>> setting in Flink's Kafka Consumer is only used for group offset fetching
>> and committing offsets back to Kafka (only for exposure purposes, not used
>> for processing guarantees).
>> The Flink Kafka Consumer uses static partition assignment on the
>> KafkaConsumer API, and not consumer group-based automatic partition
>> assignments.
>>
>> Cheers,
>> Gordon
>>
>> On Sun, Jan 27, 2019 at 12:28 AM Dawid Wysakowicz <dwysakowicz@apache.org>
>> wrote:
>>
>>> Forgot to cc Gordon :)
>>>
>>> On 23/01/2019 18:02, Avi Levi wrote:
>>> > Hi,
>>> > This quite confusing.
>>> > I submitted the same stateless job twice (actually I upload it once).
>>> > However when I place a message on kafka, it seems that both jobs
>>> > consumes it, and publish the same result (we publish the result to
>>> > other kafka topic, so I actually see the massage duplicated on kafka
>>> > ). how can it be ? both jobs are using the same group id (group id is
>>> > fixed and not generated )
>>> >
>>> > Kind regards
>>> > Avi
>>>
>>>

-- 





Regards,
Selvaraj C

Mime
View raw message