flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arvid Heise (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8716) AvroSerializer does not use schema of snapshot
Date Tue, 20 Feb 2018 13:56:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370067#comment-16370067

Arvid Heise commented on FLINK-8716:

Btw, the current behavior is that it will fail with an EOFException when a field is added.
If it was intended that the AvroSerializer should not have soft migration, it should probably
return the incompatible flag like it did before.

> AvroSerializer does not use schema of snapshot
> ----------------------------------------------
>                 Key: FLINK-8716
>                 URL: https://issues.apache.org/jira/browse/FLINK-8716
>             Project: Flink
>          Issue Type: Bug
>    Affects Versions: 1.4.0
>            Reporter: Arvid Heise
>            Priority: Major
> The new AvroSerializer stores the schema in the snapshot and uses it to validate compability.
> However, it does not use the schema of the snapshot while reading the data. This version
will fail for any change of the data layout (so it supports more or less only renaming currently).
>  [https://github.com/apache/flink/blob/f3a2197a23524048200ae2b4712d6ed833208124/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java#L265]
>  needs to use the schema from
>  [https://github.com/apache/flink/blob/f3a2197a23524048200ae2b4712d6ed833208124/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java#L188]
>  as the first parameter. Accordingly, a readSchema field need to be set
>  in #ensureCompatibility and relayed in #duplicate. Note that the readSchema is passed
as the write schema parameter to the DatumReader, as it was the schema that was used to write
the data.

This message was sent by Atlassian JIRA

View raw message