kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3988) KafkaConfigBackingStore assumes configs will be stored as schemaless maps
Date Mon, 22 Jan 2018 22:21:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335029#comment-16335029
] 

Ewen Cheslack-Postava commented on KAFKA-3988:
----------------------------------------------

This wouldn't be an issue with JsonConverter unless you are also using it with `schemas.enable`
mode, which would be unusual.

In fact, I'm going to close this now because we found that allowing flexibility in the internal
converters just caused confusion and was unnecessary. See [https://cwiki.apache.org/confluence/display/KAFKA/KIP-174+-+Deprecate+and+remove+internal+converter+configs+in+WorkerConfig] for
the KIP that is deprecating, then removing them. The configs will just always be in schemaless
JSON.

> KafkaConfigBackingStore assumes configs will be stored as schemaless maps
> -------------------------------------------------------------------------
>
>                 Key: KAFKA-3988
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3988
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 0.10.0.0
>            Reporter: Ewen Cheslack-Postava
>            Assignee: Ewen Cheslack-Postava
>            Priority: Major
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> If you use an internal key/value converter that drops schema information (as is the default
in the config files we provide since we use JsonConverter with schemas.enable=false), the
schemas we use that are structs get converted to maps since we don't know the structure to
decode them to. Because our tests run with these settings, we haven't validated that the code
works if schemas are preserved.
> When they are preserved, we'll hit an error message like this
> {quote}
> [2016-07-25 07:36:34,828] ERROR Found connector configuration (connector-test-mysql-jdbc)
in wrong format: class org.apache.kafka.connect.data.Struct (org.apache.kafka.connect.storage.KafkaConfigBackingStore:498)
> {quote}
> because the code currently checks that it is working with a map. We should actually be
checking for either a Struct or a Map. This same problem probably affects a couple of other
types of data in the same class as Connector configs, Task configs, Connect task lists, and
target states are all Structs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message