kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinoth Chandar (Jira)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-9430) Tighten up lag estimates when source topic optimization is on
Date Tue, 14 Jan 2020 17:40:00 GMT
Vinoth Chandar created KAFKA-9430:
-------------------------------------

             Summary: Tighten up lag estimates when source topic optimization is on 
                 Key: KAFKA-9430
                 URL: https://issues.apache.org/jira/browse/KAFKA-9430
             Project: Kafka
          Issue Type: Sub-task
            Reporter: Vinoth Chandar
            Assignee: Vinoth Chandar


Right now, we use _endOffsets_ of the source topic for the computation. Since the source topics
can also have user event produces, this is an over estimate

 

>From John:

For "optimized" changelogs, this will be wrong, strictly speaking, but it's an over-estimate
(which seems better than an under-estimate), and it's also still an apples-to-apples comparison,
since all replicas would use the same upper bound to compute their lags, so the "pick the
freshest" replica is still going to pick the right one. We can add a new 2.5 blocker ticket
to really fix it, and not worry about it until after this KSQL stuff is done.

 

For active: we need to use  consumed offsets and not end of source topic



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message