flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jared Stehler <jared.steh...@intellifylearning.com>
Subject class loader issues when closing streams
Date Thu, 11 Jan 2018 15:07:23 GMT
I’m seeing sporadic issues where it appears that curator (or other) user threads are left
running after a stream shutdown, and then the user class loader goes away and I get spammed
with ClassNotFoundExceptions… I’m wondering if this might have something to do with perhaps
the UserClassLoader being shut down before close is invoked on all operators?

Here’s a stack trace I see from an attempt at closing an elastic search sink:

java.lang.ClassNotFoundException: com.intellify.flink.shaded.curator.org.apache.curator.utils.CloseableUtils
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.intellify.flink.shaded.curator.org.apache.curator.ConnectionState.close(ConnectionState.java:119)
    at com.intellify.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient.close(CuratorZookeeperClient.java:227)
    at com.intellify.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl.close(CuratorFrameworkImpl.java:381)
    at com.intellify.config.ManagedCuratorFramework.reallyClose(ManagedCuratorFramework.java:48)
    at com.intellify.config.ArchaiusInitializer.close(ArchaiusInitializer.java:75)
    at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:303)
    at com.intellify.flink.shared.config.SharedIntellifyConfigProvider.close(SharedIntellifyConfigProvider.java:56)
    at com.intellify.flink.shared.config.SerializableLiveProperty.close(SerializableLiveProperty.java:68)
    at com.intellify.flink.shared.elasticsearch.LiveResolvingEs1ApiCallBridge.cleanup(LiveResolvingEs1ApiCallBridge.java:105)
    at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.close(ElasticsearchSinkBase.java:323)
    at com.intellify.flink.shared.tracer.TracingSink.close(TracingSink.java:50)
    at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:43)
    at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:117)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:446)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:351)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
    at java.lang.Thread.run(Thread.java:748)
I’m using a curator connection for archaius, and closing it in the call bridge’s cleanup
method. I’m ensuring that I’m not reaching up into the parent class loader by shading
curator and zookeeper. 

I also see the following on repeat in my task manager log:

2018-01-11 14:53:13.313 [heartbeat-filter -> input-trace-filter -> filter-inactive-ds
-> filter-duplicates (ip-10-80-53-99.us-west-2.compute.internal:2181)] WARN  c.i.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn
 - Session 0x3c00002d8a7603de for server ip-10-80-53-99.us-west-2.compute.internal/10.80.53.99:2181,
unexpected error, closing socket connection and attempting reconnect
java.lang.NoClassDefFoundError: com/intellify/flink/shaded/zookeeper/org/apache/zookeeper/proto/SetWatches
	at com.intellify.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:926)
	at com.intellify.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
	at com.intellify.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)


Does anyone have any insight into what might be happening here? Does this seem like I’m
not closing a thread properly, or something else entirely?


--
Jared Stehler
Chief Architect - Intellify Learning
o: 617.701.6330 x703




Mime
View raw message