kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ismael Juma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6598) Kafka to support using ETCD beside Zookeeper
Date Wed, 28 Mar 2018 16:57:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417741#comment-16417741

Ismael Juma commented on KAFKA-6598:

This was discussed before when "KIP-30: Allow for brokers to have plug-able consensus and
meta data storage sub systems" was proposed. It's worth reading that discussion for the concerns
with a pluggable approach.

> Kafka to support using ETCD beside Zookeeper
> --------------------------------------------
>                 Key: KAFKA-6598
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6598
>             Project: Kafka
>          Issue Type: New Feature
>          Components: clients, core
>            Reporter: Sebastian Toader
>            Priority: Major
> The current Kafka implementation is bound to {{Zookeeper}} to store its metadata for
forming a cluster of nodes (producer/consumer/broker). 
> As Kafka is becoming popular for streaming in various environments where {{Zookeeper}}
is either not easy to deploy/manage or there are better alternatives to it there is a need

> to run Kafka with other metastore implementation than {{Zookeeper}}.
> {{etcd}} can provide the same semantics as {{Zookeeper}} for Kafka and since {{etcd}}
is the favorable choice in certain environments (e.g. Kubernetes) Kafka should be able to
run with {{etcd}}.
> From the user's point of view should be straightforward to configure to use {{etcd}}
by just simply specifying a connection string that point to {{etcd}} cluster.
> To avoid introducing instability the original interfaces should be kept and only the
low level {{Zookeeper}} API calls should be replaced with \{{etcd}} API calls in case Kafka
is configured 
> to use {{etcd}}.
> On the long run (which is out of scope of this jira) there should be an abstract layer
in Kafka which then various metastore implementations would implement.

This message was sent by Atlassian JIRA

View raw message