kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5635) KIP-181 Kafka-Connect integrate with kafka ReST Proxy
Date Tue, 25 Jul 2017 17:27:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100399#comment-16100399
] 

Ewen Cheslack-Postava commented on KAFKA-5635:
----------------------------------------------

bq. Kafka connect currently uses kafka clients which directly connect to kafka brokers. 
bq. In a use case, wherein I have many kafka connect [producers] running remotely its a challenge
to configure broker information on every connect agent.

It doesn't seem like using a REST proxy would help here, you still need to configure *that*
for all workers. The common solution to simplify this is to use a load balancer with a single,
fixed address as your bootstrap servers. Then you have a single location where you need to
maintain a list of broker addresses.

bq. Also, in case of IP change [upgrade or cluster re-creation], we need to update every remote
connect configuration.

See the above solution, which also addresses this concern.

bq. If kafka connect source connectors talk to ReST endpoint then client is unaware of broker
details. This way we can transparently upgrade / re-create kafka cluster as long as ReST endpoint
remains same.

This would be unlikely to be implemented as part of the core Kafka project since REST proxies
are 3rd party. There is no standard REST proxy API for Kafka that we could rely on the protocol
for.


> KIP-181 Kafka-Connect integrate with kafka ReST Proxy
> -----------------------------------------------------
>
>                 Key: KAFKA-5635
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5635
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Dhananjay Patkar
>              Labels: features, newbie
>
> Kafka connect currently uses kafka clients which directly connect to kafka brokers. 
> In a use case, wherein I have many kafka connect [producers] running remotely its a challenge
to configure broker information on every connect agent.
> Also, in case of IP change [upgrade or cluster re-creation], we need to update every
remote connect configuration.
> If kafka connect source connectors talk to ReST endpoint then client is unaware of broker
details. This way we can transparently upgrade / re-create kafka cluster as long as ReST endpoint
remains same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message