flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Objects deserialization on Jobmanager
Date Wed, 22 Apr 2015 15:46:49 GMT
Hi Ventura!

Flink loads on startup only the Flink classes and all user code classes are
loaded dynamically. Each point when a user class is used, or a user class
object is deserialized, Flink uses the user code classloader of that
specific job.

For you as the user, this has the implication that if try to load a class
via "Class.forName", you need to supply the user code classloader as well.

You can grab the usercode class loader in various ways:
1) Inside a UDF, you can grab it by making this a RichFunction and using
"getRuntimeContext().getUsercodeClassloader()"
2) At almost any other play, you can use
"Thread.currentThread().getContextClassloader()"

Hope that helps.

Greetings,
Stephan



On Wed, Apr 22, 2015 at 4:48 PM, Till Rohrmann <trohrmann@apache.org> wrote:

> The corresponding code snippet could also help.
>
> Cheers,
>
> Till
>
> On Wed, Apr 22, 2015 at 4:45 PM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
>> Hi,
>>
>> which version of Flink are you using?
>>
>> Can you send us the complete stack trace of the error to help us
>> understand the exact location where the issue occurs?
>>
>> On Wed, Apr 22, 2015 at 4:33 PM, Ventura Del Monte <
>> venturadelmonte@gmail.com> wrote:
>>
>>> Hello, I am working on a flink-based deep learning library for my
>>> master's thesis. I am experiencing this issue at the moment: I have a java
>>> class with a transient field, so I had to write both a kryo custom
>>> serializer and a java one. The (de)serialization needs to access another
>>> object of my system, so if I run my software locally it works fine because
>>> the needed object is instantiated meanwhile it crashes when I run it in a
>>> remote environment because when the jobmanager receives the data, the
>>> object needed for the deserialization is not present in the system. Thus,
>>> my question is whether it is possible to let the jobmanager execute some
>>> user code or would it be better to edit the architecture of my system in
>>> order to avoid this kind of problem?
>>>
>>> Regards,
>>> Ventura
>>>
>>
>>
>

Mime
View raw message