kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4567) Connect Producer and Consumer ignore ssl parameters configured for worker
Date Fri, 30 Dec 2016 03:13:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15786745#comment-15786745

Ewen Cheslack-Postava commented on KAFKA-4567:

Given some future security features we may want to support (e.g. supporting different identities
for different connectors), we probably don't want to just include the worker-level security
configs into the producer & consumer. It's annoying to have to duplicate them now, but
we probably want to support more flexible combinations in the future, such as having unique
credentials for the workers (limiting the ability to, e.g., write to the config/offsets/status
topics) than those used by producers and consumers (where we may want both unique credentials
to apply ACLs and maybe support things like delegation tokens in the future).

So I think the short term solution is probably to just update the docs to clarify that you'll
currently need the settings both at the worker level and prefixed by {{producer.} and {{consumer.}}
if you're trying to use the same credentials for worker, producer, and consumer.

> Connect Producer and Consumer ignore ssl parameters configured for worker
> -------------------------------------------------------------------------
>                 Key: KAFKA-4567
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4567
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions:
>            Reporter: Sönke Liebau
>            Assignee: Ewen Cheslack-Postava
>            Priority: Minor
> When using Connect with a SSL enabled Kafka cluster, the configuration options are either
documented a bit misleading, or handled in an incorrect way.
> The documentation states the usual available SSL options (ssl.keystore.location, ssl.truststore.location,
...) and these are picked up and used for the producers and consumers that are used to communicate
with the status, offset and configs topics.
> For the producers and consumers that are used for the actual data, these parameters are
ignored as can be seen [here|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L98],
which results in plaintext communication on an SSL port, leading to an OOM exception ([KAFKA-4493|https://issues.apache.org/jira/browse/KAFKA-4493]).
> So in order to get Connect to communicate with a secured cluster you need to override
all SSL configs with the prefixes _consumer._ and _producer._ and duplicate the values already
set at a global level.
> The documentation states: 
> bq. The most critical site-specific options, such as the Kafka bootstrap servers, are
already exposed via the standard worker configuration.
> Since the address for the cluster is exposed here, I would propose that there is no reason
not to also pass the SSL parameters through to the consumers and producers, as it is clearly
intended that communication happens with the same cluster. 
> In fringe cases, these can still be overridden manually to achieve different behavior.
> I am happy to create a pull request to address this or clarify the docs, after we decide
which one is the appropriate course of action.

This message was sent by Atlassian JIRA

View raw message