kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Randall Hauch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6387) Worker's producer and consumer configs should inherit from worker configs
Date Mon, 08 Jan 2018 19:20:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316820#comment-16316820

Randall Hauch commented on KAFKA-6387:

There are a number of cases where this behavior might actually change how the producers and
consumers used for connectors are configured. Consider a worker configuration includes the
following configuration properties:


This configuration file does not define "{{consumer.max.partition.fetch.bytes}}" or "{{producer.max.partition.fetch.bytes}}",
so currently these would default to 1048576. However, after this proposed change if implemented
the "{{consumer.max.partition.fetch.bytes}}" or "{{producer.max.partition.fetch.bytes}}" values
would be the inherited value of 262144, not the default 1048576.

In short, implementing this change would break backward compatibility. We could implement
a configuration switch that controls whether the configurations are inherited, but this adds
complexity to the already-complex configuration mechanism.

> Worker's producer and consumer configs should inherit from worker configs
> -------------------------------------------------------------------------
>                 Key: KAFKA-6387
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6387
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 1.0.0
>            Reporter: Randall Hauch
>              Labels: needs-kip
> Currently, the worker configuration file defines the connection properties for the three
separate types of connections being made to the Kafka cluster:
> # the worker group membership,
> # producers for source connectors,
> # the consumers for sink connectors. 
> The configs are namespaced because to properly support things like interceptors where
the configs for 2 and 3 would conflict (same config name, different value).
> However, it would be beneficial when such control is not required for the producers and
consumers to inherit the top-level configurations yet be able to override them with the {{producer.}}
and {{consumer.}} namespaced configurations. This way the producer- and consumer-specific
configurations need only be specified if/when they need to override the top-level configurations.
This may be necessary, for example, to have different ACLs than the connector tasks compared
to the producers and consumers.
> This will require a minimal KIP to explain the new behavior. 

This message was sent by Atlassian JIRA

View raw message