kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinoth Chandar (Jira)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-8870) Prevent dirty reads of Streams state store from Interactive queries
Date Wed, 04 Sep 2019 17:36:00 GMT
Vinoth Chandar created KAFKA-8870:

             Summary: Prevent dirty reads of Streams state store from Interactive queries
                 Key: KAFKA-8870
                 URL: https://issues.apache.org/jira/browse/KAFKA-8870
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Vinoth Chandar

Today, Interactive Queries (IQ) against Streams state store could see uncommitted data, even
with EOS processing guarantees (these are actually orthogonal, but clarifying since EOS may
give the impression that everything is dandy). This is causes primarily because state updates
in rocksdb are visible even before the kafka transaction is committed. Thus, if the instance
fails, then the failed over instance will redo the uncommited old transaction and the following
could be possible during recovery,.

Value for key K can go from *V0 → V1 → V2* on active instance A, IQ reads V1, instance
A fails and any failure/rebalancing will leave the standy instance B rewinding offsets and
reprocessing, during which time IQ can again see V0 or V1 or any number of previous values
for the same key.

In this issue, we will plan work towards providing consistency for IQ, for a single row in
a single state store. i.e once a query sees V1, it can only see either V1 or V2.




This message was sent by Atlassian Jira

View raw message