kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tommy Becker (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5241) GlobalKTable does not checkpoint offsets after restoring state
Date Mon, 15 May 2017 12:01:04 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010389#comment-16010389

Tommy Becker commented on KAFKA-5241:

I have a patch for this and will take the issue, if someone can grant me permission to assign
to myself.

> GlobalKTable does not checkpoint offsets after restoring state
> --------------------------------------------------------------
>                 Key: KAFKA-5241
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5241
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions:
>            Reporter: Tommy Becker
>            Priority: Minor
> I'm experimenting with an application that uses a relatively large GlobalKTable, and
noticed that streams was not checkpointing its offsets on close(). This is because although
 {{org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl#restoreState}} updates
the checkpoint map, the actual checkpointing itself is guarded by a check that the offsets
passed from the {{GloablStateUpdateTask}} are not empty. This is frustrating because if the
topic backing the global table is both large (therefore taking a long time to restore) and
infrequently written, then streams rebuilds the table from scratch every time the application
is started.

This message was sent by Atlassian JIRA

View raw message