flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Biao Liu <mmyy1...@gmail.com>
Subject Re: Custom log appender for YARN
Date Thu, 01 Aug 2019 05:09:15 GMT
Hi Gyula,

I guess it should work if you put log appender jar under $flink/lib folder.

There are two different kinds of classloader in your case. One is for Flink
framework, the other is for user code. The framework classloader is the
parent of user classloader. The parent classloader could not find class
from sub-classloader due to the delegation model [1]. In your case, the `
ClusterEntrypoint` is in parent classloader, the log appender class in
fatjar is in sub-classloader.

So I think there are two ways to solve this.
1. Put your appender class under $flink/lib folder.
2. Try to avoid using user-defined appender for Flink framework class. (I'm
not sure it could be done or not)

You could find more informations in document [2] about Flink classloading

1. https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html

Biao /'bɪ.aʊ/

On Wed, Jul 31, 2019 at 9:21 PM Gyula Fóra <gyula.fora@gmail.com> wrote:

> Hi All!
> We are trying to configure a custom Kafka log appender for our YARN
> application and we hit the following problem.
> We included the log appender dependency in the fatjar of the application
> because in YARN that should be part of the system class path.
> However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint,
> YarnTaskExecutorRunner) these only seem to have the contents of lib on
> their classpath. Does someone have any pointers to how the classpath is
> configured for running the entrypoints?
> The exception is this:
> java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 	at java.lang.Class.forName0(Native Method)
> 	at java.lang.Class.forName(Class.java:264)
> 	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
> 	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
> 	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
> 	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
> 	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
> 	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
> 	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> 	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
> 	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
> 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
> 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
> 	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104)
> Thanks,
> Gyula

View raw message