kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Kreps (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-293) Allow to configure all broker ids at once similar to how zookeeper handles server ids
Date Fri, 09 Mar 2012 22:34:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226547#comment-13226547

Jay Kreps commented on KAFKA-293:

The current approach is, admittedly, hacky; but the zookeeper approach is equally hacky if
you ask me. I don't think it is good to put any host information in your server config.

If we want to fix this issue I think the right way to go would be to create a zk sequence
of ids. Servers would store the broker id in their data directory (maybe we could add a ${data.dir}/meta.properties
file to hold this kind of stuff). If a server starts and has no meta.properties file then
it would increment the zk id counter and take a new id and store it persistently with its
logs. On startup the broker would take its id from this file.

The reason this approach is better is because the node.id is not really configuration. That
is if you swap the node.ids for two brokers you effectively corrupt the logs in the eyes of
consumers. So the node.id is pinned to a particular set of logs. This would make any manual
borking of your node id impossible and takes the node id out of the things the user has to

> Allow to configure all broker ids at once similar to how zookeeper handles server ids
> -------------------------------------------------------------------------------------
>                 Key: KAFKA-293
>                 URL: https://issues.apache.org/jira/browse/KAFKA-293
>             Project: Kafka
>          Issue Type: Improvement
>    Affects Versions: 0.7
>            Reporter: Thomas Dudziak
> Zookeeper allows to specify all server ids in the same configuration (https://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_configuration)
which has the benefit that the configuration file is the same for all zookeeper instances.
A similar approach for Kafka would be quite useful, e.g.
> brokerid.1=<host 1>
> brokerid.2=<host 2>
> etc.
> It'd still require per-instance configuration (myid file in the zookeeper case) but that
can be created separately (e.g. by the deployment tool used).

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message