flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giriraj <graj.chau...@gmail.com>
Subject Re: Flink kafka consumers don't honor group.id
Date Tue, 03 Jul 2018 14:14:40 GMT
Hi Gordon,

Gordon:If I understood you correctly, what you are doing is, while a job
with a Kafka consumer is already running, you want to start a new job also
with a Kafka consumer as the source and uses the same group.id so that the
topic's messages are routed between the two jobs.

Is this correct? If so, could you briefly explain what your use case is and
why you want to do this?
Giriraj: You got it almost correctly. The so called "new job" above is like
a new instance of the same job. We are trying to scale the job by spawning
more instances of same job. i.e. we are abstracting the flink job as a
microservice and when load increases on this service/job, we would like to
spawn a new instance of same job/service. That is why we are expecting when
group.id is same in both the jobs, messages should get delivered only to 
one of the job consumers. I have also given thought of scaling job by
increasing parallelism(canceling and starting job with increased
parallelism). But  former approach looked cleaner,  intuitive and seamless
to us. 

I would really appreciate your thoughts about it. 

Apology for the delay in response. 


Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

View raw message