flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rinat <r.shari...@cleverdata.ru>
Subject [deserialization schema] skip data, that couldn't be properly deserialized
Date Thu, 04 Oct 2018 17:14:09 GMT
Hi mates, in accordance with the contract of org.apache.flink.formats.avro.DeserializationSchema,
it should return null value, when content couldn’t be deserialized.
But in most cases (for example org.apache.flink.formats.avro.AvroDeserializationSchema) method
fails if data is corrupted. 

We’ve implemented our own SerDe class, that returns null, if data doesn’t satisfy avro
schema, but it’s rather hard to maintain this functionality during migration to the latest
Flink version.
What do you think, maybe it’ll be useful if we will support optional skip of failed records
in avro and other Deserializers in the source code ?

Sincerely yours,
Rinat Sharipov
Software Engineer at 1DMP CORE Team

email: r.sharipov@cleverdata.ru <mailto:a.totmakov@cleverdata.ru>
mobile: +7 (925) 416-37-26

make your data clever

View raw message