kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajini Sivaram (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4631) Refresh consumer metadata more frequently for unknown subscribed topics
Date Wed, 01 Mar 2017 14:11:45 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890234#comment-15890234
] 

Rajini Sivaram commented on KAFKA-4631:
---------------------------------------

The attached PR solves the issue in the second failure in KAFKA-4779. The system test failure
was due to unavailable topics where leader is not known rather than nonexistent topics. The
PR requests metadata refresh for both unavailable and non-existent topics (See https://github.com/apache/kafka/pull/2608
for details).

> Refresh consumer metadata more frequently for unknown subscribed topics
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-4631
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4631
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>            Reporter: Jason Gustafson
>            Assignee: Matthias J. Sax
>            Priority: Minor
>             Fix For: 0.10.3.0
>
>
> By default, the consumer refreshes metadata every 5 minutes. In testing, it can often
happen that a topic is created at about the same time that the consumer is started. In the
worst case, creation finishes after the consumer fetches metadata, and the test must wait
5 minutes for the consumer to refresh metadata in order to discover the topic. To address
this problem, users can decrease the metadata refresh interval, but this means more frequent
refreshes even after all topics are known. An improvement would be to internally let the consumer
fetch metadata more frequently when the consumer encounters unknown topics. Perhaps every
5-10 seconds would be reasonable, for example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message