flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward <egb...@hotmail.com>
Subject Classloader issue with UDF's in DataStreamSource
Date Mon, 28 Aug 2017 13:53:32 GMT
I need help debugging a problem with using user defined functions in my
DataStreamSource code.

Here's the behavior:
The first time I upload my jar to the Flink cluster and submit the job, it
runs fine.
For any subsequent runs of the same job, it's giving me a NoClassDefFound
error on one of my UDF classes.
If I restart the Flink cluster, this it will again work, but only the first
time I submit the job.

I am using a customized KafkaAvroDeserializer where the reader schema is
different from the writer schema (and where that reader schema in a
generated Avro class in which is included in my uploaded jar file). If I
change my code to use the standard KafkaAvroDeserializer (i.e. no UDF's in
the DataStreamSource), it works fine, even though there are UDF's used in
other steps of my job, so the problem seems specific to DataStreamSource

Why would the classloader here not have access to all classes in my uploaded
jar file, while the classloader used in subsequent steps does have access to
that jar file? Why would it work fine the first time I upload the jar via
the Flink Dashboard, but not on subsequent executions?

Here's the exception I'm seeing:

View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Classloader-issue-with-UDF-s-in-DataStreamSource-tp15192.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

View raw message