flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tai Gordon <tzuli...@gmail.com>
Subject Re: Flink kafka group question
Date Fri, 29 Jul 2016 03:53:54 GMT

1. What Flink Kafka connector version are you using?
2. How is your non-Flink consumer fetching data from the topic? Is it using
the old SimpleConsumer, old High-Level Consumer, or the new consumer API?
3. If you are using the new consumer API, are you using
"consumer.assign(…)" or "consumer.subscribe(…)"?

Internally, the Flink Kafka connectors don’t use the consumer group
management functionality because they are using lower-level APIs
(SimpleConsumer in 0.8, and KafkaConsumer#assign(…) in 0.9) on each
parallel instance for more control on individual partition consumption. So,
essentially, the “group.id” setting in the Flink Kafka connector is only
used for committing offsets back to ZK / Kafka brokers.

>From your description I’m not entirely sure of what’s happening, so it’d be
helpful if you can provide info about the above questions so we can clarify
things further :)


On July 29, 2016 at 12:11:19 AM, 王萌 (wangm@unicall.cc) wrote:

Hi, I am using kafka in couple of programs including flink and I am quite
confused that how group.id parameter work in flink with kafka comsumer.
I have 2 consumers (one in side flink, one outside) running on the same
topic and same group.id. From my inpection, they work isolately:
If I send one message into the kafka topic, both consumer would receive it.
But acording to the kafka document, only one (same topic same group) should
be able to get the message.

And i tried to start 2 consumers both inside the flink main function, it
seems that only one consumer would be receiving all messages from kafka.

So, it's quite confused how does flink deal with this kind of situation?

Thanks a lot

View raw message