flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kien Truong <duckientru...@gmail.com>
Subject Re: NullPointerException with Avro Serializer
Date Wed, 20 Dec 2017 09:54:50 GMT
It turn out that our flink branch is out-of-date. Sorry for all the noise. :)

Regards,
Kien

⁣Sent from TypeApp ​

On Dec 20, 2017, 16:42, at 16:42, Kien Truong <duckientruong@gmail.com> wrote:
>Upon further investigation, we found out that the reason:
>
>* The cluster was started on YARN with the hadoop classpath, which
>includes Avro. Therefore, Avro's SpecificRecord class was loaded using
>sun.misc.Launcher$AppClassLoader
>
>
>* Our LteSession class was submitted with the application jar, and
>loaded with the child-first classloader
>
>* Flink check if LteSession is assignable to SpecificRecord, which
>fails.
>
>* Flink fall back to Reflection-based avro writer, which throws NPE on
>null field.
>
>If we change the classloader to parent-first, everything is ok. Now the
>question is why the default doesn't work for us.
>
>Best regards,
>Kien
>
>⁣Sent from TypeApp ​
>
>On Dec 20, 2017, 14:09, at 14:09, Kien Truong <duckientruong@gmail.com>
>wrote:
>>Hi,
>>
>>After upgrading to Flink 1.4, we encounter this exception
>>
>>Caused by: java.lang.NullPointerException: in
>>com.viettel.big4g.avro.LteSession in long null of long in field tmsi
>of
>>com.viettel.big4g.avro.LteSession
>>at
>>org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:161)
>>at
>>org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:62)
>>at
>>org.apache.flink.formats.avro.typeutils.AvroSerializer.serialize(AvroSerializer.java:132)
>>at
>>org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:176)
>>at
>>org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:48)
>>at
>>org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:54)
>>at
>>org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:93)
>>at
>>org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:114)
>>at
>>org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:89)
>>at
>>org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:84)
>>at
>>org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:102)
>>
>>
>>It seems Flink attempts to use the reflection writer instead of the
>>specific writer for this schema. This is wrong, because our LteSession
>>is an Avro object, and should use the specific writer.
>>
>>Best regards,
>>Kien
>>
>>⁣Sent from TypeApp ​

Mime
View raw message