kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <g...@confluent.io>
Subject Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector
Date Fri, 08 Sep 2017 02:04:45 GMT
I just re-read the code for the OffsetStorageWriter, and ran into this

* Note that this only provides write functionality. This is
intentional to ensure stale data is
* never read. Offset data should only be read during startup or
reconfiguration of a task. By
* always serving those requests by reading the values from the backing
store, we ensure we never
* accidentally use stale data. (One example of how this can occur: a
task is processing input
* partition A, writing offsets; reconfiguration causes partition A to
be reassigned elsewhere;
* reconfiguration causes partition A to be reassigned to this node,
but now the offset data is out
* of date). Since these offsets are created and managed by the
connector itself, there's no way
* for the offset management layer to know which keys are "owned" by
which tasks at any given
* time.

I can't figure out how the KIP avoids the stale-reads problem explained here.

Can you talk me through it? I'm cancelling my vote since right now
exposing this interface sounds risky and misleading.


On Thu, Sep 7, 2017 at 5:04 PM Gwen Shapira <gwen@confluent.io> wrote:

> +1 (binding)
> Looking forward to see how connector implementations use this in practice
> :)
> On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch <rhauch@gmail.com> wrote:
>> I'd like to open the vote for KIP-131:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector
>> Thanks to Florian for submitting the KIP and the implementation, and to
>> everyone else that helped review.
>> Best regards,
>> Randall

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message