kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-5510) Streams should commit all offsets regularly
Date Sat, 24 Jun 2017 19:29:00 GMT
Matthias J. Sax created KAFKA-5510:

             Summary: Streams should commit all offsets regularly
                 Key: KAFKA-5510
                 URL: https://issues.apache.org/jira/browse/KAFKA-5510
             Project: Kafka
          Issue Type: Bug
          Components: streams
            Reporter: Matthias J. Sax

Currently, Streams commits only offsets of partitions it did process records for. Thus, if
a partition does not have any data for longer then {{offsets.retention.minutes}} (default
1 day) the latest committed offset get's lost. On failure or restart {{auto.offset.rese}}
kicks in potentially resulting in reprocessing old data.

Thus, Streams should commit _all_ offset on a regular basis. Not sure what the overhead of
a commit is -- if it's too expensive to commit all offsets on regular commit, we could also
have a second config that specifies an "commit.all.interval".

This relates to https://issues.apache.org/jira/browse/KAFKA-3806, so we should sync to get
a solid overall solution.

This message was sent by Atlassian JIRA

View raw message