kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Per Steffensen <perst...@gmail.com>
Subject Re: [DISCUSS] KIP-381 Connect: Tell about records that had their offsets flushed in callback
Date Wed, 17 Oct 2018 15:24:04 GMT
On 17/10/2018 16.43, Ryanne Dolan wrote:
> I see, thanks.
> On the other hand, the commitRecord() callback provides the functionality
> you require in this case. In commitRecord() your SourceTask can track the
> offsets of records that have been ack'd by the producer client, and then in
> commit() you can be sure that those offsets have been flushed.
That is the trick I am currently using - more or less.
But unfortunately it does not work 100% either. It is possible that 
commitRecord() is called with a record R, and then commit() is called 
after that, without the offsets of R having been written/flushed. The 
call to commitRecord() means that the "actual data" of R has been 
send/acknowledged, but unfortunately this does not guarantee that the 
offsets of R have been written/flushed at the next commit() call

View raw message