flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benchao Li (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-19790) Writing MAP<STRING, STRING> to Kafka with JSON format produces incorrect data.
Date Wed, 04 Nov 2020 12:50:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-19790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226033#comment-17226033
] 

Benchao Li commented on FLINK-19790:
------------------------------------

Fixed via b7487bd85bedea8bd50e706c253a61ebbcf8a8bc (1.12.0)

TODO(1.11.3)

> Writing MAP<STRING, STRING> to Kafka with JSON format produces incorrect data.
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-19790
>                 URL: https://issues.apache.org/jira/browse/FLINK-19790
>             Project: Flink
>          Issue Type: Bug
>          Components: Table SQL / Ecosystem
>    Affects Versions: 1.11.2
>            Reporter: Fabian Hueske
>            Assignee: Benchao Li
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 1.12.0, 1.11.3
>
>
> Running the following SQL script writes incorrect data to Kafka:
> {code:java}
> CREATE TEMPORARY TABLE tmp_1 (m MAP<String, String>) WITH (
>   'connector' = 'kafka',
>   'format' = 'json',
>   'properties.bootstrap.servers' = '...',
>   'properties.group.id' = '...',
>   'topic' = 'tmp-1'
> );
> CREATE TEMPORARY TABLE gen (k STRING, v STRING) WITH (
>   'connector' = 'datagen'
> );
> CREATE TEMPORARY VIEW gen_short AS
> SELECT SUBSTR(k, 0, 4) AS k, SUBSTR(v, 0, 4) AS v FROM gen;
> INSERT INTO tmp_1
> SELECT MAP[k, v] FROM gen_short; {code}
> Printing the content of the {{tmp-1}} topics results in the following output:
> {code:java}
> $ kafka-console-consumer --bootstrap-server ... --from-beginning --topic tmp-1 | head
-n 5
> {"m":{"8a93":"6102"}}
> {"m":{"8a93":"6102","7922":"f737"}}
> {"m":{"8a93":"6102","7922":"f737","9b63":"15b0"}}
> {"m":{"8a93":"6102","7922":"f737","9b63":"15b0","c38b":"b55c"}}
> {"m":{"8a93":"6102","7922":"f737","9b63":"15b0","c38b":"b55c","222c":"f3e2"}}
> {code}
> As you can see, the map is not correctly encoded as JSON and written to Kafka.
> I've run the query with the Blink planner with object reuse and operator pipelining disabled.
> Writing with Avro works as expected.
> Hence I assume that the JSON encoder/serializer reuses the Map object when encoding the
JSON.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message