flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Zagrebin <azagrebin.apa...@gmail.com>
Subject Re: about Kafka sink and 2PC function
Date Fri, 18 Oct 2019 12:46:51 GMT
Hi,

This is the contract of 2PC transactions.
Multiple commit retries should result in only one commit which actually happens in the external
system.
The external system has to support deduplication of committed transactions, e.g. by some unique
id.

Best,
Andrey

> On 10 Oct 2019, at 07:15, 121476721@qq.com wrote:
> 
> After reading about FlinkKafkaProducer011 and 2PC function in FLINK, I know, 
> when snapshotState(),
> preCommit currentTransaction.
> add <currentTransaction, newTransaction> to the State.
> when Checkpoint done and notifyCheckpointComplete(),
> producer will commit currentTransaction to brokers.
> when initializeState(),
> restore from State.
> commit currentTransaction and abort newTransaction.
> And I have one question, what happens if program fails after notifyCheckpointComplete()
done?
> As my opinion, when it recovers, it will re-commit what has committed which results duplicate.
> 
> 121476721@qq.com <mailto:121476721@qq.com>

Mime
View raw message