kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edvard Poliakov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6002) Kafka Connect Transform transforming JSON string into actual object
Date Mon, 02 Oct 2017 18:05:00 GMT
Edvard Poliakov created KAFKA-6002:

             Summary: Kafka Connect Transform transforming JSON string into actual object
                 Key: KAFKA-6002
                 URL: https://issues.apache.org/jira/browse/KAFKA-6002
             Project: Kafka
          Issue Type: Improvement
          Components: KafkaConnect
            Reporter: Edvard Poliakov
            Priority: Minor

My colleague and I have been working on a new Transform, that takes a JSON string and transforms
it into an actual object, like this:

  "a" : "{\"b\": 23}"
  "a" : {
       "b" : 23

There is no robust way of building a Schema from a JSON object itself, as it can be something
like an empty array or a null, that doesn't provide any info on the schema of the object.
So I see two options here.

1. For a transform to take in schema as a transform parameter. The problem I found with this
is that it is not clear what JSON schema specification should be used for this? I assume it
would be reasonable to use http://json-schema.org/, but it doesn't seem that Kafka Connect
supports it currently, moreover reading through JsonConverter class in Kafka Connect, I am
not able to understand what spec does the Json Schema have that is used in that class, for
example {{asConnectSchema}} method on {{JsonConverte}}.

2. On each object received, keep updating the schema, but I can't see a standard and robust
way of handling edge cases.

I am happy to create a pull request for this transform, if we can agree on something here.

This message was sent by Atlassian JIRA

View raw message