kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Randall Hauch (Jira)" <j...@apache.org>
Subject [jira] [Assigned] (KAFKA-10600) Connect adds error to property in validation result if connector does not define the property
Date Fri, 16 Oct 2020 14:18:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Randall Hauch reassigned KAFKA-10600:

    Fix Version/s: 2.6.1
         Reviewer: Konstantine Karantasis
         Assignee: Randall Hauch
       Resolution: Fixed

Connect should not always add an error to configuration values in validation results that
don't have a `ConfigKey` defined in the connector's `ConfigDef`, and any errors on such configuration
values included by the connector should be counted in the total number of errors. Added more
unit tests for `AbstractHerder.generateResult(...)`.

> Connect adds error to property in validation result if connector does not define the
> ---------------------------------------------------------------------------------------------
>                 Key: KAFKA-10600
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10600
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions:,, 1.0.0, 1.1.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0,
2.5.0, 2.6.0
>            Reporter: Randall Hauch
>            Assignee: Randall Hauch
>            Priority: Major
>             Fix For: 2.7.0, 2.5.2, 2.6.1
> Kafka Connect's {{AbstractHerder.generateResult(...)}} method is responsible for taking
the result of a {{Connector.validate(...)}} call and constructing the {{ConfigInfos}} object
that is then mapped to the JSON representation.
> As this method (see [code|https://github.com/apache/kafka/blob/1f8ac6e6fee3aa404fc1a4c01ac2e0c48429a306/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L504-L507])
iterates over the {{ConfigKey}} objects in the connector's {{ConfigDef}} and the {{ConfigValue}}
objects returned by the {{Connector.validate(...)}} method, this method adds an error message
to any {{ConfigValue}} whose {{configValue.name()}} does not correspond to a {{ConfigKey}}
in the connector's {{ConfigDef}}. 
> {code}
>             if (!configKeys.containsKey(configName)) {
>                 configValue.addErrorMessage("Configuration is not defined: " + configName);
>                 configInfoList.add(new ConfigInfo(null, convertConfigValue(configValue,
>             }
> {code}
> Interestingly, these errors are not included in the total error count of the response.
Is that intentional??
> This behavior does not allow connectors to report validation errors against extra properties
not defined in the connector's {{ConfigDef}}. 
> Consider a connector that allows arbitrary properties with some prefix (e.g., {{connection.*}})
to be included and used in the connector properties. One example is to supply additional properties
to a JDBC connection, where the connector may not be able to know these "additional properties"
in advance because the connector either works with multiple JDBC drivers or the connection
properties allowed by a JDBC driver are many and/or vary over different JDBC driver versions
or server versions.
> Such "additional properties" are not prohibited by Connect API, yet if a connector implementation
chooses to include any such additional properties in the {{Connector.validate(...)}} result
(whether or not the corresponding {{ConfigValue}} has an error) then Connect will always add
the following error to that property. 
> {quote}
> Configuration is not defined: <additionalPropertyName>
> {quote}
> This code was in the release of Kafka via the [PR|https://github.com/apache/kafka/pull/964]
for KAFKA-3315, which is one of the tasks that implemented [KIP-26|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767]
for Kafka Connect (approved and partially added in There is no mention of "validation"
in KIP-26 nor any followup KIP (that I can find).
> I can kind of imagine the original thought process: any user-supplied property that is
not defined by a {{ConfigDef}} is inherently an error. However, this assumption is not matched
by any mention in the Connect API, documentation, or one of Connect's KIP.
> IMO, this is a bug in the {{AbstractHerder}} that over-constrains the connector properties
to be only those defined in the connector's {{ConfigDef}}.
> Quite a few connectors already support additional properties, and it's perhaps only by
chance that this happens to work: 
> * If a connector does not override {{Connector.validate(...)}}, extra properties are
not validated and therefore are not included in the resulting {{Config}} response with one
{{ConfigValue}} per property defined in the connector's {{ConfigDef}}.
> * If a connector does override {{Connector.validate(...)}} and includes in the {{Config}}
response a {{ConfigValue}} for the any additional properties, the {{AbstractHerder.generateResults(...)}}
method does add the error but does not include this error in the error count, which is actually
used to determine if there are any validation problems before starting/updating the connector.
> I propose that the {{AbstractHerder.generateResult(...)}} method be changed to not add
any additional error message to the validation result, and to properly handle all {{ConfigValue}}
objects regardless of whether there is a corresponding {{ConfigKey}} in the connector's {{ConfigDef}}.

This message was sent by Atlassian Jira

View raw message