flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Avoiding Dynamic Classloading
Date Wed, 22 Nov 2017 11:43:05 GMT
Hi,

Yes, if I remember correctly, this was changed in 1.2 to always include the user-jar in the
system classloader on YARN. With Flink 1.4 we are changing the user-code classloader to load
classes from the user-jar first (child-first classloading) by default so a lot of the comments
on avoiding dynamic classloading are not relevant anymore then.

Side note, you should only worry about those issues if you're seeing problems that hint at
that, such as ClassNotFound exceptions or exceptions about missing methods.

Best,
Aljoscha

> On 20. Nov 2017, at 22:04, Chan, Regina <Regina.Chan@gs.com> wrote:
> 
> Hi,
>  
> I was reading that I should avoid using dynamic classloading and so copy the job’s
jar into the /lib directory (RE: below)
>  
> 1.     How can I confirm that the jar was copied over? I only see the following below:
>  
> 2017-11-20 15:36:52,724 INFO  org.apache.flink.yarn.Utils                           
       - Copying from file:/local/data/scratch/tmp/p2epdlsu/chregi/flink-1.2.0/lib to hdfs://d173636/user/delp_prod/.flink/application_1511197407590_58493/lib
<hdfs://d173636/user/delp_prod/.flink/application_1511197407590_58493/lib>
> 2017-11-20 15:37:04,644 INFO  org.apache.flink.yarn.Utils                           
       - Copying from file:/local/data/scratch/tmp/p2epdlsu/chregi/flink-1.2.0/lib/flink-dist_2.10-1.2.0.jar
to hdfs://d173636/user/delp_prod/.flink/application_1511197407590_58493/flink-dist_2.10-1.2.0.jar
<hdfs://d173636/user/delp_prod/.flink/application_1511197407590_58493/flink-dist_2.10-1.2.0.jar>
> 2017-11-20 15:37:06,634 INFO  org.apache.flink.yarn.Utils                           
       - Copying from /home/p2epdlsu/datalake-cdc-prod/etc/flink/conf/flink-conf.yaml tohdfs://d173636/user/delp_prod/.flink/application_1511197407590_58493/flink-conf.yaml
<hdfs://d173636/user/delp_prod/.flink/application_1511197407590_58493/flink-conf.yaml>
>  
> 2.     I also saw this ticket https://issues.apache.org/jira/browse/FLINK-4913 <https://issues.apache.org/jira/browse/FLINK-4913>
and was wondering whether this is orthogonal to the dynamic loading and having to put my jar
in the lib directory. Or should this be handled by default already.
> 
> 3.     I start flink from a globally mounted/shared copy that I don’t have write access
to. I can’t easily put jars in that lib folder. For the same reason I shouldn’t modify
the global copy of the bin/config.sh. Is there a way to configure where flink picks up the
lib folder from?
>  
> Thanks!
>  
> 
>  
>  
> Avoiding Dynamic Classloading
> 
> All components (JobManger, TaskManager, Client, ApplicationMaster, …) log their classpath
setting on startup. They can be found as part of the environment information at the beginning
of the log.
> 
> When running a setup where the Flink JobManager and TaskManagers are exclusive to one
particular job, one can put JAR files directly into the /lib folder to make sure they are
part of the classpath and not loaded dynamically.
> 
> It usually works to put the job’s JAR file into the /lib directory. The JAR will be
part of both the classpath (the AppClassLoader) and the dynamic class loader (FlinkUserCodeClassLoader).
Because the AppClassLoader is the parent of the FlinkUserCodeClassLoader (and Java loads parent-first),
this should result in classes being loaded only once.
> 
> For setups where the job’s JAR file cannot be put to the /lib folder (for example because
the setup is a session that is used by multiple jobs), it may still be possible to put common
libraries to the /lib folder, and avoid dynamic class loading for those.
> 
>  
>  
> Regina Chan
> Goldman Sachs – Enterprise Platforms, Data Architecture
> 30 Hudson Street, 37th floor | Jersey City, NY 07302 (  (212) 902-5697


Mime
View raw message