kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neha Narkhede (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-788) Periodic refresh of topic metadata on the producer doesn't include all topics
Date Tue, 05 Mar 2013 19:12:12 GMT
Neha Narkhede created KAFKA-788:

             Summary: Periodic refresh of topic metadata on the producer doesn't include all
                 Key: KAFKA-788
                 URL: https://issues.apache.org/jira/browse/KAFKA-788
             Project: Kafka
          Issue Type: Bug
          Components: producer 
    Affects Versions: 0.8
            Reporter: Neha Narkhede
            Assignee: Jun Rao

We added a patch to the producer to refresh the metadata for all topics periodically. However,
the producer only does this for the topics in the last batch. But some topics sent by the
producer could be low throughput and might not be present in every batch. If we bounce the
cluster or if brokers fail and leaders change, the metadata for those low throughput topic
is not refreshed by this periodic topic metadata request. The next produce request for those
topics have to fail and then a separate metadata request needs to be reissued to handle the
produce request. This is especially a problem for the migration tool. So even if the producer
had a chance to refresh the metadata when the leader changed, it throws LeaderNotAvailableExceptions
much later when it sends a request for that topic. 

I propose we just fetch data for all topics sent by the producer in the periodic refresh of
topic metadata

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message