flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9444) KafkaAvroTableSource failed to work for map and array fields
Date Tue, 05 Jun 2018 18:07:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502220#comment-16502220
] 

ASF GitHub Bot commented on FLINK-9444:
---------------------------------------

Github user tragicjun commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6082#discussion_r193168591
  
    --- Diff: flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroRecordClassConverter.java
---
    @@ -73,9 +75,37 @@ private AvroRecordClassConverter() {
     			final GenericTypeInfo<?> genericTypeInfo = (GenericTypeInfo<?>) extracted;
     			if (genericTypeInfo.getTypeClass() == Utf8.class) {
     				return BasicTypeInfo.STRING_TYPE_INFO;
    +			} else if (genericTypeInfo.getTypeClass() == Map.class) {
    +				// avro map key is always string
    +				return Types.MAP(Types.STRING,
    +					convertPrimitiveType(schema.getValueType().getType()));
    --- End diff --
    
    I've implemented a reflection version, which now supports record type within map/array.


> KafkaAvroTableSource failed to work for map and array fields
> ------------------------------------------------------------
>
>                 Key: FLINK-9444
>                 URL: https://issues.apache.org/jira/browse/FLINK-9444
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector, Table API &amp; SQL
>    Affects Versions: 1.6.0
>            Reporter: Jun Zhang
>            Priority: Blocker
>              Labels: patch
>             Fix For: 1.6.0
>
>         Attachments: flink-9444.patch
>
>
> When some Avro schema has map/array fields and the corresponding TableSchema declares *MapTypeInfo/ListTypeInfo* for
these fields, an exception will be thrown when registering the *KafkaAvroTableSource*, complaining
like:
> Exception in thread "main" org.apache.flink.table.api.ValidationException: Type Map<String,
Integer> of table field 'event' does not match with type GenericType<java.util.Map>
of the field 'event' of the TableSource return type.
>  at org.apache.flink.table.api.ValidationException$.apply(exceptions.scala:74)
>  at org.apache.flink.table.sources.TableSourceUtil$$anonfun$validateTableSource$1.apply(TableSourceUtil.scala:92)
>  at org.apache.flink.table.sources.TableSourceUtil$$anonfun$validateTableSource$1.apply(TableSourceUtil.scala:71)
>  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>  at org.apache.flink.table.sources.TableSourceUtil$.validateTableSource(TableSourceUtil.scala:71)
>  at org.apache.flink.table.plan.schema.StreamTableSourceTable.<init>(StreamTableSourceTable.scala:33)
>  at org.apache.flink.table.api.StreamTableEnvironment.registerTableSourceInternal(StreamTableEnvironment.scala:124)
>  at org.apache.flink.table.api.TableEnvironment.registerTableSource(TableEnvironment.scala:438)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message