flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Does FlinkKafkaConsumer010 care about consumer group?
Date Wed, 19 Jul 2017 07:36:56 GMT
Hi!

The only occasions which the consumer group is used is:
1. When committing offsets back to Kafka. Since Flink 1.3, this can be disabled completely
(both when checkpointing is enabled or disabled). See [1] on details about that.
2. When starting fresh (not starting from some savepoint), if you choose to use GROUP_OFFSETS
as the start position, then the consumer group would also be used. If starting from a savepoint,
then this is irrelevant. See [2].

Note that it actually isn’t used in any critical paths for Flink’s exactly-once processing
guarantees, or partition to source parallel instance assignments.

So, the desired behavior in which you described is exactly the expected behavior for the Flink
Kafka Consumer.

Cheers,
Gordon

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/kafka.html#kafka-consumers-start-position-configuration

On 19 July 2017 at 3:23:01 PM, Moiz Jinia (moiz.jinia@gmail.com) wrote:

Below is a plan for downtime-free upgrade of a Flink job. The downstream  
consumer of the Flink job is duplicate proof.  

Scenario 1 -  
1. Start Flink job A with consumer group G1 (12 slot job)  
2. While job A is running, take a savepoint AS.  
3. Start newer version of Flink job A' from savepoint AS with consumer group  
*G1* (12 slot job again)  
4. Stop job A.  

Scenario 2 -  
1. Start Flink job A with consumer group G1 (12 slot job)  
2. While job A is running, take a savepoint AS.  
3. Start newer version of Flink job A' from savepoint AS with consumer group  
*G2* (12 slot job again)  
4. Stop job A  

Does it matter what consumer group job A' uses? The desired behavior is that  
during the window when both A and A' are running, all messages should go to  
both jobs. (And of course I want that job A' should start consuming from the  
offsets in the savepoint and not the earliest).  






--  
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Does-FlinkKafkaConsumer010-care-about-consumer-group-tp14323.html
 
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
 

Mime
View raw message