kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <g...@confluent.io>
Subject Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector
Date Fri, 08 Sep 2017 02:04:45 GMT
I just re-read the code for the OffsetStorageWriter, and ran into this
comment:

* Note that this only provides write functionality. This is
intentional to ensure stale data is
* never read. Offset data should only be read during startup or
reconfiguration of a task. By
* always serving those requests by reading the values from the backing
store, we ensure we never
* accidentally use stale data. (One example of how this can occur: a
task is processing input
* partition A, writing offsets; reconfiguration causes partition A to
be reassigned elsewhere;
* reconfiguration causes partition A to be reassigned to this node,
but now the offset data is out
* of date). Since these offsets are created and managed by the
connector itself, there's no way
* for the offset management layer to know which keys are "owned" by
which tasks at any given
* time.


I can't figure out how the KIP avoids the stale-reads problem explained here.

Can you talk me through it? I'm cancelling my vote since right now
exposing this interface sounds risky and misleading.


Gwen


On Thu, Sep 7, 2017 at 5:04 PM Gwen Shapira <gwen@confluent.io> wrote:

> +1 (binding)
>
> Looking forward to see how connector implementations use this in practice
> :)
>
> On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch <rhauch@gmail.com> wrote:
>
>> I'd like to open the vote for KIP-131:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector
>>
>> Thanks to Florian for submitting the KIP and the implementation, and to
>> everyone else that helped review.
>>
>> Best regards,
>>
>> Randall
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message