kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6190) GlobalKTable never finishes restoring when consuming transactional messages
Date Wed, 08 Nov 2017 23:40:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244941#comment-16244941
] 

ASF GitHub Bot commented on KAFKA-6190:
---------------------------------------

GitHub user alexjg opened a pull request:

    https://github.com/apache/kafka/pull/4197

    KAFKA-6190 GlobalKTable never finishes restoring when consuming transactional messages

    Calculate offset using consumer.position() in GlobalStateManagerImp#restoreState

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/alexjg/kafka 0.11.0

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/4197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4197
    
----
commit d96aac2de211ee4592e795b07d059dd4dea20f95
Author: Alex Good <alexjsgood@gmail.com>
Date:   2017-11-08T23:26:09Z

    Calculate offset using consumer.position() in GlobalStateManagerImpl#restoreState

----


> GlobalKTable never finishes restoring when consuming transactional messages
> ---------------------------------------------------------------------------
>
>                 Key: KAFKA-6190
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6190
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>         Environment: Linux
>            Reporter: Alex Good
>
> When creating a GlobalKTable from a topic that contains messages that were produced in
a transaction the GlobalStreamThread never finishes restoring the table. This appears to be
because the `GlobalStateManagerImpl#restoreState` method fails to take into account the transaction
markers in it's calculation of it's offset when reading messages and so never reaches the
high watermark for the topic it is restoring.
> To demonstrate the issue produce a few messages in a transaction to a topic, then attempt
to restore a GlobalKTable from that topic, the store will never complete restoring.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message