kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert V (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-7556) KafkaConsumer.beginningOffsets does not return actual first offsets
Date Fri, 26 Oct 2018 14:39:00 GMT
Robert V created KAFKA-7556:
-------------------------------

             Summary: KafkaConsumer.beginningOffsets does not return actual first offsets
                 Key: KAFKA-7556
                 URL: https://issues.apache.org/jira/browse/KAFKA-7556
             Project: Kafka
          Issue Type: Bug
          Components: clients, consumer
    Affects Versions: 1.0.0
            Reporter: Robert V


h2. Description of the problem

The method `org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets` claims in its
Javadoc documentation that it would 'Get the first offset for the given partitions.'.

I used it with a compacted topic, and it always returned offset 0 for all partitions.
Not sure if using a compacted topic actually matters, but I'm enclosing this information anyway.

Given a Kafka topic with retention set, and old log files being removed as a result of that,
the effective start offset of those partitions move further; it simply will be greater than
offset 0.

However, calling the `beginningOffsets` method always returns offset 0 as the first offset.

In contrast, when the method `org.apache.kafka.clients.consumer.KafkaConsumer.offsetsForTimes`
is called with a timestamp of 0L (UNIX epoch 1st Jan, 1970), it correctly returns the effective
start offsets for each partitions.

Output of using `org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets`:

 
{code:java}
List((test.topic-87,Some(0)), (test.topic-54,Some(0)), (test.topic-21,Some(0)), (test.topic-79,Some(0)),
(test.topic-46,Some(0)), (test.topic-13,Some(0)), (test.topic-70,Some(0)), (test.topic-37,Some(0)),
(test.topic-12,Some(0)), (test.topic-95,Some(0)), (test.topic-62,Some(0)), (test.topic-29,Some(0)),
(test.topic-4,Some(0)), (test.topic-88,Some(0)), (test.topic-55,Some(0)), (test.topic-22,Some(0)),
(test.topic-80,Some(0)), (test.topic-47,Some(0)), (test.topic-14,Some(0)), (test.topic-71,Some(0)),
(test.topic-38,Some(0)), (test.topic-5,Some(0)), (test.topic-96,Some(0)), (test.topic-63,Some(0)),
(test.topic-30,Some(0)), (test.topic-56,Some(0)), (test.topic-23,Some(0)), (test.topic-89,Some(0)),
(test.topic-48,Some(0)), (test.topic-15,Some(0)), (test.topic-81,Some(0)), (test.topic-72,Some(0)),
(test.topic-39,Some(0)), (test.topic-6,Some(0)), (test.topic-64,Some(0)), (test.topic-31,Some(0)),
(test.topic-97,Some(0)), (test.topic-24,Some(0)), (test.topic-90,Some(0)), (test.topic-57,Some(0)),
(test.topic-16,Some(0)), (test.topic-82,Some(0)), (test.topic-49,Some(0)), (test.topic-40,Some(0)),
(test.topic-7,Some(0)), (test.topic-73,Some(0)), (test.topic-32,Some(0)), (test.topic-98,Some(0)),
(test.topic-65,Some(0)), (test.topic-91,Some(0)), (test.topic-58,Some(0)), (test.topic-25,Some(0)),
(test.topic-83,Some(0)), (test.topic-50,Some(0)), (test.topic-17,Some(0)), (test.topic-8,Some(0)),
(test.topic-74,Some(0)), (test.topic-41,Some(0)), (test.topic-0,Some(0)), (test.topic-99,Some(0)),
(test.topic-66,Some(0)), (test.topic-33,Some(0)), (test.topic-92,Some(0)), (test.topic-59,Some(0)),
(test.topic-26,Some(0)), (test.topic-84,Some(0)), (test.topic-51,Some(0)), (test.topic-18,Some(0)),
(test.topic-75,Some(0)), (test.topic-42,Some(0)), (test.topic-9,Some(0)), (test.topic-67,Some(0)),
(test.topic-34,Some(0)), (test.topic-1,Some(0)), (test.topic-85,Some(0)), (test.topic-60,Some(0)),
(test.topic-27,Some(0)), (test.topic-77,Some(0)), (test.topic-52,Some(0)), (test.topic-19,Some(0)),
(test.topic-76,Some(0)), (test.topic-43,Some(0)), (test.topic-10,Some(0)), (test.topic-93,Some(0)),
(test.topic-68,Some(0)), (test.topic-35,Some(0)), (test.topic-2,Some(0)), (test.topic-86,Some(0)),
(test.topic-53,Some(0)), (test.topic-28,Some(0)), (test.topic-78,Some(0)), (test.topic-45,Some(0)),
(test.topic-20,Some(0)), (test.topic-69,Some(0)), (test.topic-44,Some(0)), (test.topic-11,Some(0)),
(test.topic-94,Some(0)), (test.topic-61,Some(0)), (test.topic-36,Some(0)), (test.topic-3,Some(0)))
{code}
Output of using `org.apache.kafka.clients.consumer.KafkaConsumer.offsetsForTimes`:
{code:java}
List((test.topic-87,Some(289)), (test.topic-54,Some(45420)), (test.topic-21,Some(63643)),
(test.topic-79,Some(380750)), (test.topic-46,Some(266379)), (test.topic-13,Some(98512)), (test.topic-70,Some(118522)),
(test.topic-37,Some(763)), (test.topic-12,Some(61108)), (test.topic-95,Some(847647)), (test.topic-62,Some(68299)),
(test.topic-29,Some(548361)), (test.topic-4,Some(144855)), (test.topic-88,Some(107831)), (test.topic-55,Some(129241)),
(test.topic-22,Some(563095)), (test.topic-80,Some(534859)), (test.topic-47,Some(71608)), (test.topic-14,Some(80962)),
(test.topic-71,Some(83607)), (test.topic-38,Some(166460)), (test.topic-5,Some(294307)), (test.topic-96,Some(483237)),
(test.topic-63,Some(441051)), (test.topic-30,Some(151727)), (test.topic-56,Some(171045)),
(test.topic-23,Some(103736)), (test.topic-89,Some(276851)), (test.topic-48,Some(696518)),
(test.topic-15,Some(82049)), (test.topic-81,Some(426329)), (test.topic-72,Some(76346)), (test.topic-39,Some(39325)),
(test.topic-6,Some(501911)), (test.topic-64,Some(517845)), (test.topic-31,Some(68886)), (test.topic-97,Some(378159)),
(test.topic-24,Some(185533)), (test.topic-90,Some(54403)), (test.topic-57,Some(154119)), (test.topic-16,Some(91612)),
(test.topic-82,Some(158657)), (test.topic-49,Some(67356)), (test.topic-40,Some(133092)), (test.topic-7,Some(178568)),
(test.topic-73,Some(1897104)), (test.topic-32,Some(291600)), (test.topic-98,Some(813386)),
(test.topic-65,Some(97431)), (test.topic-91,Some(302858)), (test.topic-58,Some(416483)), (test.topic-25,Some(85250)),
(test.topic-83,Some(426855)), (test.topic-50,Some(151059)), (test.topic-17,Some(71425)), (test.topic-8,Some(386542)),
(test.topic-74,Some(79210)), (test.topic-41,Some(84127)), (test.topic-0,Some(56920)), (test.topic-99,Some(34036)),
(test.topic-66,Some(285773)), (test.topic-33,Some(627169)), (test.topic-92,Some(67471)), (test.topic-59,Some(398275)),
(test.topic-26,Some(167519)), (test.topic-84,Some(1820740)), (test.topic-51,Some(201574)),
(test.topic-18,Some(447521)), (test.topic-75,Some(101415)), (test.topic-42,Some(102338)),
(test.topic-9,Some(1012234)), (test.topic-67,Some(157516)), (test.topic-34,Some(18917)), (test.topic-1,Some(114164)),
(test.topic-85,Some(59734)), (test.topic-60,Some(1395047)), (test.topic-27,Some(567605)),
(test.topic-77,Some(45907)), (test.topic-52,Some(212559)), (test.topic-19,Some(98667)), (test.topic-76,Some(304288)),
(test.topic-43,Some(130635)), (test.topic-10,Some(26730)), (test.topic-93,Some(545451)), (test.topic-68,Some(160989)),
(test.topic-35,Some(694191)), (test.topic-2,Some(79696)), (test.topic-86,Some(142042)), (test.topic-53,Some(44083)),
(test.topic-28,Some(97500)), (test.topic-78,Some(95872)), (test.topic-45,Some(346044)), (test.topic-20,Some(166317)),
(test.topic-69,Some(143188)), (test.topic-44,Some(870214)), (test.topic-11,Some(182125)),
(test.topic-94,Some(1022343)), (test.topic-61,Some(1204397)), (test.topic-36,Some(172640)),
(test.topic-3,Some(98632)))
{code}
h2. Proposed fix
 * If the intention is to return what the beginning offset ever was, the Javadoc documentation
should be explicit about it and another method should be added to query the effective start
offsets.
 * Alternatively, the method should return the actual start offset of the partitions at the
time of calling this method. Which, for topics with retention policy set, is not equal to
what the start offsets were before removing log files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message