flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Change Kafka cluster without loosing the Flink's state
Date Thu, 14 Sep 2017 10:10:31 GMT
Hi Konstantin,

After migrating and connecting to the new Kafka cluster, do you want the Kafka consumer to
start fresh without any partition offset state (and therefore will re-establish its partition-to-subtask
assignments), while keeping all other operator state in the pipeline intact?
If so, that is definitely possible. Simply explicitly set a new different operator UID for
the FlinkKafkaConsumer. When the savepoint is restored, the consumer will not be restored
with any state since it will be recognized by Flink as a new operator in the pipeline without
any state.

Cheers,
Gordon

On 14 September 2017 at 11:56:38 AM, kla (lalafaryan@gmail.com) wrote:

Hi guys, 

We have a running apache flink streaming job which interacts with apache 
kafka (consumer and producer). 
Now we would like to change the kafka cluster without loosing Flink's state. 

Is it possible to do it ? If yes, what is the right way to do it ? 

Thanks in advance! 
Best, 
Konstantin 



-- 
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ 

Mime
View raw message