kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajini Sivaram (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5537) Subscribe Earliest is not working as in 0.10.2.1
Date Wed, 05 Jul 2017 08:42:02 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074428#comment-16074428
] 

Rajini Sivaram commented on KAFKA-5537:
---------------------------------------

[~michael.andre.pearce] I have run your test and it passes consistently for me if I increase
the first sleep (before produce) from 4000 to 8000. Your logs for 0.11.0 show that you are
producing before partitions are assigned. With offsets reset to "latest", this will mean that
the messages produced before assignment are not consumed.

>From your logs for 0.11.0:
{quote}
2017-06-30 23:02:15 INFO  AbstractCoordinator:597 - Discovered coordinator localhost:52196
(id: 2147483646 rack: null) for group test-group.
017-06-30 23:02:15 INFO  AbstractCoordinator:597 - Discovered coordinator localhost:52196
(id: 2147483646 rack: null) for group test-group.
...
017-06-30 23:02:18 TRACE KafkaProducer:740 - Sending record ProducerRecord(topic=topic, partition=null,
headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value=[B@62379589, timestamp=null)
with callback null to topic topic partition 0
...
017-06-30 23:02:21 INFO  ConsumerCoordinator:262 - Setting newly assigned partitions [] for
group test-group
2017-06-30 23:02:21 INFO  ConsumerCoordinator:262 - Setting newly assigned partitions (topic-0)
for group test-group
{quote}

Messages are produced 4 seconds after the consumer was created. But at this point, rebalancing
hasn't completed and no partitions have been assigned to the consumers. When the partitions
are assigned 3 seconds later, consumers start consuming from the "latest" offset at point,
ignoring already produced messages. With {{AUTO_OFFSET_RESET_CONFIG=earliest}}, this would
have worked.

With lower {{group.initial.rebalance.delay.ms}}, the test works since rebalance completes
sooner. When your test was waiting for 60 seconds, can you check if the wait was before producing
messages when offset reset strategy is {{latest}}?

> Subscribe Earliest is not working as in 0.10.2.1
> ------------------------------------------------
>
>                 Key: KAFKA-5537
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5537
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.11.0.0
>            Reporter: Michael Andre Pearce
>            Priority: Critical
>         Attachments: kafka_0.10.2.1.log, kafka_0.11.0.0.log, KafkaSub.java, KafkaSubLatest.java
>
>
> We have seen issue with subscription where auto offset when set to earliest (and also
latest) does not behave the same as in 0.10.2.1 release.
> We have managed to create a repeatable test for this, which passes when pointing to 0.10.2.1
broker.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message