flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Smirnov <alexander.smirn...@gmail.com>
Subject Re: Kafka exceptions in Flink log file
Date Thu, 05 Apr 2018 13:16:19 GMT
Hi Timo,

it is the latest released version - 1.4.2

This only happens when a job falls into a restart loop and stays in it for
20 minutes or so.
Looks like for each restart, Flink loads classes anew, but previously
loaded classes are not garbage collected for some reason (still referenced?)

Very soon, JVM runs out of Metaspace and this error occurs.

Thank you,
Alex



On Tue, Apr 3, 2018 at 4:24 PM Timo Walther <twalthr@apache.org> wrote:

> Hi Alex,
>
> which version of Flink are you running? There were some class loading
> issues with Kafka recently. I would try it with the newest Flink
> version. Otherwise ClassNotFoundException usually indicates that
> something is wrong with your dependencies. Maybe you can share your
> pom.xml with us.
>
> Regards,
> Timo
>
> Am 02.04.18 um 13:32 schrieb Alexander Smirnov:
> > I see a lot of messages in flink log like below. What's the cause?
> >
> >
> > 02 Apr 2018 04:09:13,554 ERROR
> > org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
> > kafka producer I/O thread:
> > org.apache.kafka.common.KafkaException: Error registering mbean
> >
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
> >         at
> >
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:163)
> >         at
> >
> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:81)
> >         at
> > org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:504)
> >         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:255)
> >         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:240)
> >         at
> >
> org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:811)
> >         at
> >
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:353)
> >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:326)
> >         at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
> >         at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
> >         at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
> >         at java.lang.Thread.run(Thread.java:748)
> > Caused by: javax.management.InstanceAlreadyExistsException:
> >
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
> >         at
> > com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> >         at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> >         at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> >         at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> >         at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> >         at
> >
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> >         at
> >
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:161)
> >         ... 11 more
> > 02 Apr 2018 04:09:13,673 ERROR
> > org.apache.kafka.common.utils.KafkaThread - Uncaught exception in
> > kafka-producer-network-thread | producer-3:
> > java.lang.NoClassDefFoundError: org/apache/kafka/clients/NetworkClient$1
> >         at
> >
> org.apache.kafka.clients.NetworkClient.processDisconnection(NetworkClient.java:583)
> >         at
> >
> org.apache.kafka.clients.NetworkClient.handleDisconnections(NetworkClient.java:705)
> >         at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:443)
> >         at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
> >         at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
> >         at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.kafka.clients.NetworkClient$1
> >         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >         at
> >
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >         ... 6 more
> >
> >
> > Thank you,
> > Alex
>
>
>

Mime
View raw message