kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neha Narkhede (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (KAFKA-238) add a getTopicMetaData method in broker and expose it to producer
Date Thu, 05 Jan 2012 19:08:39 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180715#comment-13180715
] 

Neha Narkhede edited comment on KAFKA-238 at 1/5/12 7:07 PM:
-------------------------------------------------------------

As part of adding this getTopicMetaData and changing the producer to use it, we can get rid
of the zookeeper client in the producer al together. 

Here is how the load balancing would work -

1. On startup, the producer can invoke the getTopicMetaData API and cache the topic and partition
information locally.
2. If the leader of a partition goes down, some send() request from the producer would fail.
At this point, the producer will perform actions in step 1 again. And then, retry the send
request.

Since mastership change happens pretty rarely, this approach seems reasonable, given that
we can get rid of zookeeper client from the producer. 

Now, for step 2 to work correctly with the async producer, we need to first resolve KAFKA-233.
That will take care of moving the load balancing logic to the end of the queue. So, actions
in step 2 can be part of the EventHandler
                
      was (Author: nehanarkhede):
    As part of adding this getTopicMetaData and changing the producer to use it, we can get
rid of the zookeeper client in the producer al together. 

Here is how the load balancing would work -

1. On startup, the producer can invoke the getTopicMetaData API and cache the topic and partition
information locally.
2. If the leader of a partition goes down, some send() request from the producer would fail.
At this point, the producer will perform actions in step 1 again.

Since mastership change happens pretty rarely, this approach seems reasonable, given that
we can get rid of zookeeper client from the producer. 

Now, for step 2 to work correctly with the async producer, we need to first resolve KAFKA-233.
That will take care of moving the load balancing logic to the end of the queue. So, actions
in step 2 can be part of the EventHandler
                  
> add a getTopicMetaData method in broker and expose it to producer 
> ------------------------------------------------------------------
>
>                 Key: KAFKA-238
>                 URL: https://issues.apache.org/jira/browse/KAFKA-238
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: core
>            Reporter: Jun Rao
>
> We need a way to propagate the leader and the partition information to the producer so
that it can do load balancing and semantic partitioning. One way to do that is to have the
producer get the information from ZK directly. This means that the producer needs to maintain
a ZK session and has to subscribe to watchers, which can be complicated. An alternative approach
is to have the following api on the broker.
> TopicMetaData getTopicMetaData(String: topic)
> TopicMetaData {
>   Array[PartitionMetaData]: partitionsMetaData
> }
> PartitionMetaData {
>   Int: partitionId
>   String: leaderHostname
>   Int: leaderPort
> }
> Using this api, the producer can get the metadata about a topic during initial startup
or leadership change of a partition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message