Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Date: Thu, 5 Jan 2012 19:18:39 +0000 (UTC)
From: "Jay Kreps (Commented) (JIRA)" <jira@apache.org>
To: kafka-dev@incubator.apache.org
Message-ID: 
 <1320361913.11130.1325791119468.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1800535561.10805.1325788839341.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (KAFKA-238) add a getTopicMetaData method in
 broker and expose it to producer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/KAFKA-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180736#comment-13180736 ] 

Jay Kreps commented on KAFKA-238:
---------------------------------

One general recommendation I have is that we think about making this as general as possible. The way we are doing APIs is fairly high overhead, and the general difficulty of doing client upgrades means we really want to have fewer, more general apis.

Questions: 
Shouldn't we also give back the host/port info for replicas? Might there be a case where it is better to read from replicas?
Should this be per-topic or should we just get all metadata?
Also, should we consider other metadata? For example the other metadata we have on a per-partition basis is node id, total data size, number of segments, beginning offset and date for each segment. This would make it possible to get rid of the getOffsetsBefore api as that would be a special case of this more general metadata api.

Related thought: currently we maintain a custom object for each request. This is very good for fetch and produce which need efficiency for the message sets. It is kind of a hassle for this case. An alternative would be to just send a single string field containing the metadata as json, which would make evolution easier for this kind of performance-insensitive api. On the other hand since it is just one api object maybe the cost of a JSON parser jar and non-uniformity with the other apis isn't worth it.
                
> add a getTopicMetaData method in broker and expose it to producer 
> ------------------------------------------------------------------
>
>                 Key: KAFKA-238
>                 URL: https://issues.apache.org/jira/browse/KAFKA-238
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: core
>            Reporter: Jun Rao
>
> We need a way to propagate the leader and the partition information to the producer so that it can do load balancing and semantic partitioning. One way to do that is to have the producer get the information from ZK directly. This means that the producer needs to maintain a ZK session and has to subscribe to watchers, which can be complicated. An alternative approach is to have the following api on the broker.
> TopicMetaData getTopicMetaData(String: topic)
> TopicMetaData {
>   Array[PartitionMetaData]: partitionsMetaData
> }
> PartitionMetaData {
>   Int: partitionId
>   String: leaderHostname
>   Int: leaderPort
> }
> Using this api, the producer can get the metadata about a topic during initial startup or leadership change of a partition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira