kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4159) Allow overriding producer & consumer properties at the connector level
Date Wed, 22 Mar 2017 18:11:41 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936835#comment-15936835

Ewen Cheslack-Postava commented on KAFKA-4159:

[~sjdurfey] Allowing configs like this is a public API change, so it'd need a KIP. You can
find the instructions for a KIP here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
It might sound like quite a bit of overhead, but for simple changes (especially additions
with no compatibility concerns) it's usually very straightforward. Once that's through a PR
could be merged.

That said, I'm curious about the use case. Why would multiple connectors need to share the
same consumer group? Why couldn't you just change the number of tasks for the connector? This
seems like quite an unusual use case, so I'm not sure it's something that makes sense in the
framework. More generally, I personally tend to view the producer and consumer (and their
configs) as implementation details -- I think exposing all configs might be masking other
issues that could be better addressed by the framework. To me, the motivation section of a
KIP would be important in this case. (But that's just my viewpoint, I know other people have
asked about this in the past.)

> Allow overriding producer & consumer properties at the connector level
> ----------------------------------------------------------------------
>                 Key: KAFKA-4159
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4159
>             Project: Kafka
>          Issue Type: New Feature
>          Components: KafkaConnect
>            Reporter: Shikhar Bhushan
>            Assignee: Stephen Durfey
> As an example use cases, overriding a sink connector's consumer's partition assignment

This message was sent by Atlassian JIRA

View raw message