kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Gustafson (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-5731) Connect WorkerSinkTask out of order offset commit can lead to inconsistent state
Date Fri, 11 Aug 2017 23:19:00 GMT
Jason Gustafson created KAFKA-5731:
--------------------------------------

             Summary: Connect WorkerSinkTask out of order offset commit can lead to inconsistent
state
                 Key: KAFKA-5731
                 URL: https://issues.apache.org/jira/browse/KAFKA-5731
             Project: Kafka
          Issue Type: Bug
          Components: KafkaConnect
            Reporter: Jason Gustafson
             Fix For: 1.0.0


In Connect's WorkerSinkTask, we do sequence number validation to ensure that offset commits
are handled in the right order (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L199).


Unfortunately, for asynchronous commits, the {{lastCommittedOffsets}} field is overridden
regardless of this sequence check as long as the response had no error (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L284).


Hence if we get an out of order commit, then the internal state will be inconsistent. To fix
this, we should only override {{lastCommittedOffsets}} after sequence validation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message