crail-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Crespi <david.cre...@storedgesystems.com>
Subject RE: Setting up storage class 1 and 2
Date Fri, 28 Jun 2019 14:44:38 GMT
This morning I only started the rdma datanode, and only enabled the namenode with the rdma storage Tier.

It looks like it found the right combination now for IP/port, but it still hangs at the exact same place.



Could this be the same problem as the other one I’ve been having (daemonization of a thread)?  I’ve included

a stack trace of this hang as well (below).



19/06/28 07:38:21 INFO disni: got event type + UNKNOWN, srcAddress /192.168.3.13:53911, dstAddress /192.168.3.100:50020

19/06/28 07:38:21 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.3.9:57008) with ID 2

19/06/28 07:38:21 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.3.10:60204) with ID 3

19/06/28 07:38:21 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.3.12:43740) with ID 1

19/06/28 07:38:21 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.3.11:39758) with ID 0

19/06/28 07:38:21 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.3.8:52590) with ID 4

19/06/28 07:38:21 DEBUG DefaultTopologyMapper: Got a request for 192.168.3.9

19/06/28 07:38:21 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.3.9:33673 with 366.3 MB RAM, BlockManagerId(2, 192.168.3.9, 33673, None)

19/06/28 07:38:21 DEBUG DefaultTopologyMapper: Got a request for 192.168.3.12

19/06/28 07:38:21 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.3.12:46629 with 366.3 MB RAM, BlockManagerId(1, 192.168.3.12, 46629, None)

19/06/28 07:38:21 DEBUG DefaultTopologyMapper: Got a request for 192.168.3.10

19/06/28 07:38:21 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.3.10:41079 with 366.3 MB RAM, BlockManagerId(3, 192.168.3.10, 41079, None)

19/06/28 07:38:21 DEBUG DefaultTopologyMapper: Got a request for 192.168.3.11

19/06/28 07:38:21 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.3.11:44679 with 366.3 MB RAM, BlockManagerId(0, 192.168.3.11, 44679, None)

19/06/28 07:38:21 DEBUG DefaultTopologyMapper: Got a request for 192.168.3.8

19/06/28 07:38:21 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.3.8:45991 with 366.3 MB RAM, BlockManagerId(4, 192.168.3.8, 45991, None)

19/06/28 07:38:30 DEBUG Client: IPC Client (1998371610) connection to NameNode-1/192.168.3.7:54310 from hduser: closed

19/06/28 07:38:30 DEBUG Client: IPC Client (1998371610) connection to NameNode-1/192.168.3.7:54310 from hduser: stopped, remaining connections 0





jstack 712

2019-06-28 07:41:54

Full thread dump OpenJDK 64-Bit Server VM (25.212-b03 mixed mode):



"Attach Listener" #132 daemon prio=9 os_prio=0 tid=0x00007fe630001000 nid=0x43f waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"SparkUI-131" #131 daemon prio=5 os_prio=0 tid=0x00007fe460001000 nid=0x3b4 waiting on condition [0x00007fe6911c8000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c0487220> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)

        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-8" #128 daemon prio=5 os_prio=0 tid=0x00007fe4fc00f800 nid=0x3ae runnable [0x00007fe44d8d2000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c048cf28> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c048e028> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c048df50> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-7" #127 daemon prio=5 os_prio=0 tid=0x00007fe4fc00e000 nid=0x3ad runnable [0x00007fe44d9d3000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c0490888> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c1782ae8> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1782a10> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-6" #126 daemon prio=5 os_prio=0 tid=0x00007fe4fc00c000 nid=0x3ac runnable [0x00007fe44fad6000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c1785348> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c1786448> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1786370> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-5" #125 daemon prio=5 os_prio=0 tid=0x00007fe4fc00a800 nid=0x3ab runnable [0x00007fe44fbd7000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c1788ca8> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c1825b38> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1825a60> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"Thread-19" #124 daemon prio=5 os_prio=0 tid=0x00007fe4a81b4800 nid=0x3aa runnable [0x00007fe454106000]

   java.lang.Thread.State: RUNNABLE

        at com.ibm.disni.verbs.impl.NativeDispatcher._getCmEvent(Native Method)

        at com.ibm.disni.verbs.impl.RdmaCmNat.getCmEvent(RdmaCmNat.java:193)

        at com.ibm.disni.verbs.RdmaEventChannel.getCmEvent(RdmaEventChannel.java:75)

        at com.ibm.disni.RdmaCmProcessor.run(RdmaCmProcessor.java:69)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-4" #123 daemon prio=5 os_prio=0 tid=0x00007fe4fc008800 nid=0x3a9 runnable [0x00007fe49075d000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c1828398> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c1829498> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c18293c0> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"threadDeathWatcher-6-1" #122 daemon prio=1 os_prio=0 tid=0x00007fe464003800 nid=0x3a8 waiting on condition [0x00007fe49185f000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

        at java.lang.Thread.sleep(Native Method)

        at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-3" #121 daemon prio=5 os_prio=0 tid=0x00007fe4fc007800 nid=0x3a7 runnable [0x00007fe491960000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c1940730> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c1941830> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1941758> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-2" #120 daemon prio=5 os_prio=0 tid=0x00007fe4fc005800 nid=0x3a6 runnable [0x00007fe491a61000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c1928738> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c1929838> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1929760> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #117 daemon prio=5 os_prio=0 tid=0x00007fe474163000 nid=0x3a3 in Object.wait() [0x00007fe6901be000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000ff70c888> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000000ff70c888> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)

        at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063)

        at java.lang.Thread.run(Thread.java:748)



"element-tracking-store-worker" #116 daemon prio=5 os_prio=0 tid=0x00007fe478012800 nid=0x3a2 waiting on condition [0x00007fe6904bf000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c025d728> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"spark-listener-group-executorManagement" #102 daemon prio=5 os_prio=0 tid=0x00007fe6fc6ed800 nid=0x3a1 waiting on condition [0x00007fe6905c0000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c0461920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:97)

        at org.apache.spark.scheduler.AsyncEventQueue$$Lambda$528/1418119693.apply$mcJ$sp(Unknown Source)

        at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)

        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)

        at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)

        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:83)

        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2$$Lambda$527/1354319042.apply$mcV$sp(Unknown Source)

        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)

        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:83)



"spark-listener-group-appStatus" #25 daemon prio=5 os_prio=0 tid=0x00007fe6fc6ec000 nid=0x3a0 waiting on condition [0x00007fe6906c1000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04624e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:97)

        at org.apache.spark.scheduler.AsyncEventQueue$$Lambda$528/1418119693.apply$mcJ$sp(Unknown Source)

        at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)

        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)

        at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)

        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:83)

        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2$$Lambda$527/1354319042.apply$mcV$sp(Unknown Source)

        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)

        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:83)



"context-cleaner-periodic-gc" #115 daemon prio=5 os_prio=0 tid=0x00007fe6fcb6a000 nid=0x39f waiting on condition [0x00007fe6907c2000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000ff758a28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"Spark Context Cleaner" #114 daemon prio=5 os_prio=0 tid=0x00007fe6fcb68000 nid=0x39e in Object.wait() [0x00007fe6908c3000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000000ff6f09c8> (a java.lang.ref.ReferenceQueue$Lock)

        at org.apache.spark.ContextCleaner.$anonfun$keepCleaning$1(ContextCleaner.scala:181)

        at org.apache.spark.ContextCleaner$$Lambda$523/1657875753.apply$mcV$sp(Unknown Source)

        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)

        at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:179)

        at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)



"shuffle-server-5-1" #113 daemon prio=5 os_prio=0 tid=0x00007fe6ff4d3000 nid=0x39d runnable [0x00007fe690bc4000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c046ecf8> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c046ed10> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c046ecb0> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-client-1-1" #112 daemon prio=5 os_prio=0 tid=0x00007fe498005800 nid=0x39b runnable [0x00007fe690ec5000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c0464cf0> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c0464d08> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c0464ca8> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"appclient-registration-retry-thread" #110 daemon prio=5 os_prio=0 tid=0x00007fe5b4003800 nid=0x399 waiting on condition [0x00007fe6910c7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c050cd28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"driver-revive-thread" #108 daemon prio=5 os_prio=0 tid=0x00007fe5c8001800 nid=0x397 waiting on condition [0x00007fe6912c9000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c048a1f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dag-scheduler-event-loop" #107 daemon prio=5 os_prio=0 tid=0x00007fe6ff4b6800 nid=0x396 in Object.wait() [0x00007fe6913c9000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000ff7b2150> (a org.apache.crail.storage.rdma.client.RdmaStoragePassiveEndpoint)

        at java.lang.Object.wait(Object.java:502)

        at com.ibm.disni.RdmaEndpoint.connect(RdmaEndpoint.java:128)

        - locked <0x00000000ff7b2150> (a org.apache.crail.storage.rdma.client.RdmaStoragePassiveEndpoint)

        at org.apache.crail.storage.rdma.client.RdmaStoragePassiveGroup.createEndpoint(RdmaStoragePassiveGroup.java:68)

        at org.apache.crail.storage.rdma.client.RdmaStoragePassiveGroup.createEndpoint(RdmaStoragePassiveGroup.java:51)

        at org.apache.crail.storage.rdma.RdmaStorageClient.createEndpoint(RdmaStorageClient.java:83)

        at org.apache.crail.utils.EndpointCache$StorageEndpointCache.getDataEndpoint(EndpointCache.java:130)

        - locked <0x00000000ff7b2258> (a java.lang.Object)

        at org.apache.crail.utils.EndpointCache.getDataEndpoint(EndpointCache.java:69)

        at org.apache.crail.core.CoreStream.prepareAndTrigger(CoreStream.java:230)

        at org.apache.crail.core.CoreStream.dataOperation(CoreStream.java:142)

        at org.apache.crail.core.CoreOutputStream.write(CoreOutputStream.java:63)

        at org.apache.crail.core.DirectoryOutputStream.writeRecord(DirectoryOutputStream.java:50)

        at org.apache.crail.core.CoreDataStore.getSyncOperation(CoreDataStore.java:647)

        at org.apache.crail.core.CoreDataStore._delete(CoreDataStore.java:335)

        at org.apache.crail.core.DeleteNodeFuture.process(CoreMetaDataOperation.java:210)

        at org.apache.crail.core.DeleteNodeFuture.process(CoreMetaDataOperation.java:196)

        at org.apache.crail.core.CoreMetaDataOperation.get(CoreMetaDataOperation.java:84)

        at org.apache.spark.storage.CrailDispatcher.org$apache$spark$storage$CrailDispatcher$$init(CrailDispatcher.scala:136)

        at org.apache.spark.storage.CrailDispatcher$.get(CrailDispatcher.scala:662)

        - locked <0x00000000ff9a53a0> (a java.lang.Object)

        at org.apache.spark.shuffle.crail.CrailShuffleManager.registerShuffle(CrailShuffleManager.scala:52)

        at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:94)

        at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:87)

        at org.apache.spark.rdd.RDD.$anonfun$dependencies$2(RDD.scala:240)

        at org.apache.spark.rdd.RDD$$Lambda$805/2019083900.apply(Unknown Source)

        at scala.Option.getOrElse(Option.scala:138)

        at org.apache.spark.rdd.RDD.dependencies(RDD.scala:238)

        at org.apache.spark.scheduler.DAGScheduler.getShuffleDependencies(DAGScheduler.scala:512)

        at org.apache.spark.scheduler.DAGScheduler.getOrCreateParentStages(DAGScheduler.scala:461)

        at org.apache.spark.scheduler.DAGScheduler.createResultStage(DAGScheduler.scala:448)

        at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:962)

        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2067)

        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)

        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)

        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)



"netty-rpc-env-timeout" #106 daemon prio=5 os_prio=0 tid=0x00007fe4b400c000 nid=0x395 waiting on condition [0x00007fe6916cb000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04bedd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"Timer-1" #105 daemon prio=5 os_prio=0 tid=0x00007fe6ff46c000 nid=0x394 in Object.wait() [0x00007fe6917cc000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000c054c030> (a java.util.TaskQueue)

        at java.lang.Object.wait(Object.java:502)

        at java.util.TimerThread.mainLoop(Timer.java:526)

        - locked <0x00000000c054c030> (a java.util.TaskQueue)

        at java.util.TimerThread.run(Timer.java:505)



"Timer-0" #104 daemon prio=5 os_prio=0 tid=0x00007fe6ff46a000 nid=0x393 in Object.wait() [0x00007fe6918cd000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000c052b400> (a java.util.TaskQueue)

        at java.lang.Object.wait(Object.java:502)

        at java.util.TimerThread.mainLoop(Timer.java:526)

        - locked <0x00000000c052b400> (a java.util.TaskQueue)

        at java.util.TimerThread.run(Timer.java:505)



"heartbeat-receiver-event-loop-thread" #103 daemon prio=5 os_prio=0 tid=0x00007fe5d0003800 nid=0x392 waiting on condition [0x00007fe691acf000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c05324c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-99-acceptor-3@162ca734-Spark@7a8fa7ef{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}" #99 daemon prio=3 os_prio=0 tid=0x00007fe6ff3c5800 nid=0x38f waiting for monitor entry [0x00007fe691bd0000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)

        - waiting to lock <0x00000000c053a468> (a java.lang.Object)

        at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)

        at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-98-acceptor-2@51df0104-Spark@7a8fa7ef{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}" #98 daemon prio=3 os_prio=0 tid=0x00007fe6ff3c4000 nid=0x38e waiting for monitor entry [0x00007fe691cd1000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)

        - waiting to lock <0x00000000c053a468> (a java.lang.Object)

        at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)

        at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-97-acceptor-1@62473a9b-ServerConnector@7a8fa7ef{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}" #97 daemon prio=3 os_prio=0 tid=0x00007fe6ff3c2000 nid=0x38d waiting for monitor entry [0x00007fe691dd2000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)

        - waiting to lock <0x00000000c053a468> (a java.lang.Object)

        at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)

        at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-96-acceptor-0@74484716-ServerConnector@7a8fa7ef{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}" #96 daemon prio=3 os_prio=0 tid=0x00007fe6ff3c0800 nid=0x38c runnable [0x00007fe691ed3000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)

        - locked <0x00000000c053a468> (a java.lang.Object)

        at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)

        at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-95" #95 daemon prio=5 os_prio=0 tid=0x00007fe6ff3be800 nid=0x38b runnable [0x00007fe691fd4000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c054a378> (a sun.nio.ch.Util$3)

        - locked <0x00000000c054a388> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c054a330> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)

        at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)

        at org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-94" #94 daemon prio=5 os_prio=0 tid=0x00007fe6ff3bd000 nid=0x38a runnable [0x00007fe6920d5000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c0496930> (a sun.nio.ch.Util$3)

        - locked <0x00000000c0496940> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c04968e8> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)

        at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)

        at org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-93" #93 daemon prio=5 os_prio=0 tid=0x00007fe6ff3bb800 nid=0x389 runnable [0x00007fe6921d6000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c0552628> (a sun.nio.ch.Util$3)

        - locked <0x00000000c0552638> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c05525e0> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)

        at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)

        at org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-92" #92 daemon prio=5 os_prio=0 tid=0x00007fe6ff3b8000 nid=0x388 runnable [0x00007fe6922d7000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c055a5f8> (a sun.nio.ch.Util$3)

        - locked <0x00000000c055a608> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c055a5b0> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)

        at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)

        at org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)

        at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"RemoteBlock-temp-file-clean-thread" #91 daemon prio=5 os_prio=0 tid=0x00007fe6ff2b5000 nid=0x387 in Object.wait() [0x00007fe6925d8000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000000c0562538> (a java.lang.ref.ReferenceQueue$Lock)

        at org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager.org$apache$spark$storage$BlockManager$RemoteBlockDownloadFileManager$$keepCleaning(BlockManager.scala:1724)

        at org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager$$anon$2.run(BlockManager.scala:1692)



"map-output-dispatcher-7" #90 daemon prio=5 os_prio=0 tid=0x00007fe6ff25f800 nid=0x386 waiting on condition [0x00007fe6928d9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c054c920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-6" #89 daemon prio=5 os_prio=0 tid=0x00007fe6ff25d800 nid=0x385 waiting on condition [0x00007fe6929da000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c054c920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-5" #88 daemon prio=5 os_prio=0 tid=0x00007fe6ff25c000 nid=0x384 waiting on condition [0x00007fe692adb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c054c920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-4" #87 daemon prio=5 os_prio=0 tid=0x00007fe6ff262000 nid=0x383 waiting on condition [0x00007fe692bdc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c054c920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-3" #86 daemon prio=5 os_prio=0 tid=0x00007fe6ff261000 nid=0x382 waiting on condition [0x00007fe692cdd000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c054c920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-2" #85 daemon prio=5 os_prio=0 tid=0x00007fe6ff252000 nid=0x381 waiting on condition [0x00007fe692dde000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c054c920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-1" #84 daemon prio=5 os_prio=0 tid=0x00007fe6ff263000 nid=0x380 waiting on condition [0x00007fe692edf000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c054c920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-0" #83 daemon prio=5 os_prio=0 tid=0x00007fe6ff256000 nid=0x37f waiting on condition [0x00007fe692fe0000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c054c920> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-1" #82 daemon prio=5 os_prio=0 tid=0x00007fe6ff213000 nid=0x37e runnable [0x00007fe6930e1000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c048a730> (a io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c048a748> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c048a6e8> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-55" #81 daemon prio=5 os_prio=0 tid=0x00007fe6fefb6000 nid=0x37d waiting on condition [0x00007fe6935f4000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-54" #80 daemon prio=5 os_prio=0 tid=0x00007fe6fefb4000 nid=0x37c waiting on condition [0x00007fe6936f5000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-53" #79 daemon prio=5 os_prio=0 tid=0x00007fe6fefb2000 nid=0x37b waiting on condition [0x00007fe6937f6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-52" #78 daemon prio=5 os_prio=0 tid=0x00007fe6fefb0800 nid=0x37a waiting on condition [0x00007fe6938f7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-51" #77 daemon prio=5 os_prio=0 tid=0x00007fe6fefae800 nid=0x379 waiting on condition [0x00007fe6939f8000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-50" #76 daemon prio=5 os_prio=0 tid=0x00007fe6fefac800 nid=0x378 waiting on condition [0x00007fe693af9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-49" #75 daemon prio=5 os_prio=0 tid=0x00007fe6fefab000 nid=0x377 waiting on condition [0x00007fe693bfa000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-48" #74 daemon prio=5 os_prio=0 tid=0x00007fe6fefa9800 nid=0x376 waiting on condition [0x00007fe693cfb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-47" #73 daemon prio=5 os_prio=0 tid=0x00007fe6fefa7800 nid=0x375 waiting on condition [0x00007fe693dfc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-46" #72 daemon prio=5 os_prio=0 tid=0x00007fe6fefa6000 nid=0x374 waiting on condition [0x00007fe693efd000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-45" #71 daemon prio=5 os_prio=0 tid=0x00007fe6fefa4000 nid=0x373 waiting on condition [0x00007fe693ffe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-44" #70 daemon prio=5 os_prio=0 tid=0x00007fe6fefa2800 nid=0x372 waiting on condition [0x00007fe6981ef000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-43" #69 daemon prio=5 os_prio=0 tid=0x00007fe6fefa1000 nid=0x371 waiting on condition [0x00007fe6982f0000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-42" #68 daemon prio=5 os_prio=0 tid=0x00007fe6fef9f000 nid=0x370 waiting on condition [0x00007fe6983f1000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-41" #67 daemon prio=5 os_prio=0 tid=0x00007fe6fef9d800 nid=0x36f waiting on condition [0x00007fe6984f2000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-40" #66 daemon prio=5 os_prio=0 tid=0x00007fe6fef9b800 nid=0x36e waiting on condition [0x00007fe6985f3000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-39" #65 daemon prio=5 os_prio=0 tid=0x00007fe6fef9a000 nid=0x36d waiting on condition [0x00007fe6986f4000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-38" #64 daemon prio=5 os_prio=0 tid=0x00007fe6fef98000 nid=0x36c waiting on condition [0x00007fe6987f5000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-37" #63 daemon prio=5 os_prio=0 tid=0x00007fe6fef96800 nid=0x36b waiting on condition [0x00007fe6988f6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-36" #62 daemon prio=5 os_prio=0 tid=0x00007fe6fef95000 nid=0x36a waiting on condition [0x00007fe6989f7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-35" #61 daemon prio=5 os_prio=0 tid=0x00007fe6fef93000 nid=0x369 waiting on condition [0x00007fe698af8000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-34" #60 daemon prio=5 os_prio=0 tid=0x00007fe6fef91800 nid=0x368 waiting on condition [0x00007fe698bf9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-33" #59 daemon prio=5 os_prio=0 tid=0x00007fe6fef8f800 nid=0x367 waiting on condition [0x00007fe698cfa000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-32" #58 daemon prio=5 os_prio=0 tid=0x00007fe6fef8e000 nid=0x366 waiting on condition [0x00007fe698dfb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-31" #57 daemon prio=5 os_prio=0 tid=0x00007fe6fef8c000 nid=0x365 waiting on condition [0x00007fe698efc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-30" #56 daemon prio=5 os_prio=0 tid=0x00007fe6fef8a800 nid=0x364 waiting on condition [0x00007fe698ffd000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-29" #55 daemon prio=5 os_prio=0 tid=0x00007fe6fef89000 nid=0x363 waiting on condition [0x00007fe6a01ab000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-28" #54 daemon prio=5 os_prio=0 tid=0x00007fe6fef87000 nid=0x362 waiting on condition [0x00007fe6a02ac000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-27" #53 daemon prio=5 os_prio=0 tid=0x00007fe6fef85800 nid=0x361 waiting on condition [0x00007fe6a03ad000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-26" #52 daemon prio=5 os_prio=0 tid=0x00007fe6fef83800 nid=0x360 waiting on condition [0x00007fe6a04ae000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-25" #51 daemon prio=5 os_prio=0 tid=0x00007fe6fef82000 nid=0x35f waiting on condition [0x00007fe6a05af000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-24" #50 daemon prio=5 os_prio=0 tid=0x00007fe6fef80800 nid=0x35e waiting on condition [0x00007fe6a06b0000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-23" #49 daemon prio=5 os_prio=0 tid=0x00007fe6fef7e800 nid=0x35d waiting on condition [0x00007fe6a07b1000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-22" #48 daemon prio=5 os_prio=0 tid=0x00007fe6fef7d000 nid=0x35c waiting on condition [0x00007fe6a08b2000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-21" #47 daemon prio=5 os_prio=0 tid=0x00007fe6fef7b000 nid=0x35b waiting on condition [0x00007fe6a09b3000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-20" #46 daemon prio=5 os_prio=0 tid=0x00007fe6fef79800 nid=0x35a waiting on condition [0x00007fe6a0ab4000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-19" #45 daemon prio=5 os_prio=0 tid=0x00007fe6fef77800 nid=0x359 waiting on condition [0x00007fe6a0bb5000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-18" #44 daemon prio=5 os_prio=0 tid=0x00007fe6fef76000 nid=0x358 waiting on condition [0x00007fe6a0cb6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-17" #43 daemon prio=5 os_prio=0 tid=0x00007fe6fef74800 nid=0x357 waiting on condition [0x00007fe6a0db7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-16" #42 daemon prio=5 os_prio=0 tid=0x00007fe6fef72800 nid=0x356 waiting on condition [0x00007fe6a0eb8000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-15" #41 daemon prio=5 os_prio=0 tid=0x00007fe6fef71000 nid=0x355 waiting on condition [0x00007fe6a0fb9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-14" #40 daemon prio=5 os_prio=0 tid=0x00007fe6fef6f000 nid=0x354 waiting on condition [0x00007fe6a10ba000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-13" #39 daemon prio=5 os_prio=0 tid=0x00007fe6fef6d800 nid=0x353 waiting on condition [0x00007fe6a11bb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-12" #38 daemon prio=5 os_prio=0 tid=0x00007fe6fef6b800 nid=0x352 waiting on condition [0x00007fe6a12bc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-11" #37 daemon prio=5 os_prio=0 tid=0x00007fe6fef6a000 nid=0x351 waiting on condition [0x00007fe6a13bd000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-10" #36 daemon prio=5 os_prio=0 tid=0x00007fe6fef68800 nid=0x350 waiting on condition [0x00007fe6a14be000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-9" #35 daemon prio=5 os_prio=0 tid=0x00007fe6fef66800 nid=0x34f waiting on condition [0x00007fe6a15bf000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-8" #34 daemon prio=5 os_prio=0 tid=0x00007fe6fef65000 nid=0x34e waiting on condition [0x00007fe6a16c0000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-7" #33 daemon prio=5 os_prio=0 tid=0x00007fe6fef63000 nid=0x34d waiting on condition [0x00007fe6a17c1000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-6" #32 daemon prio=5 os_prio=0 tid=0x00007fe6fef61800 nid=0x34c waiting on condition [0x00007fe6a18c2000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-5" #31 daemon prio=5 os_prio=0 tid=0x00007fe6fef5f800 nid=0x34b waiting on condition [0x00007fe6a19c3000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-4" #30 daemon prio=5 os_prio=0 tid=0x00007fe6fef5e000 nid=0x34a waiting on condition [0x00007fe6a1ac4000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-3" #29 daemon prio=5 os_prio=0 tid=0x00007fe6fef5a000 nid=0x349 waiting on condition [0x00007fe6a1bc5000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-2" #28 daemon prio=5 os_prio=0 tid=0x00007fe6fef58800 nid=0x348 waiting on condition [0x00007fe6a1cc6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-1" #27 daemon prio=5 os_prio=0 tid=0x00007fe6fef57000 nid=0x347 waiting on condition [0x00007fe6a1dc7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-0" #26 daemon prio=5 os_prio=0 tid=0x00007fe6fef56000 nid=0x346 waiting on condition [0x00007fe6a252c000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c045d790> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007fe6fc12e800 nid=0x341 runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007fe6fc123800 nid=0x340 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007fe6fc121800 nid=0x33f waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007fe6fc11f800 nid=0x33e waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007fe6fc11d800 nid=0x33d waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007fe6fc11b800 nid=0x33c waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007fe6fc119800 nid=0x33b waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007fe6fc117000 nid=0x33a waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007fe6fc115000 nid=0x339 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007fe6fc113000 nid=0x338 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007fe6fc110800 nid=0x337 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007fe6fc106800 nid=0x336 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007fe6fc104800 nid=0x335 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fe6fc100000 nid=0x334 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fe6fc0fe800 nid=0x333 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fe6fc0fb800 nid=0x332 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fe6fc0f9800 nid=0x331 runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fe6fc0d2800 nid=0x330 in Object.wait() [0x00007fe6a92d1000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000c001f4a8> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000000c001f4a8> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)

        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)



"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fe6fc0d0000 nid=0x32f in Object.wait() [0x00007fe6a93d2000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000c002ead8> (a java.lang.ref.Reference$Lock)

        at java.lang.Object.wait(Object.java:502)

        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)

        - locked <0x00000000c002ead8> (a java.lang.ref.Reference$Lock)

        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)



"main" #1 prio=5 os_prio=0 tid=0x00007fe6fc013000 nid=0x307 waiting on condition [0x00007fe7050e9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000ff90fd18> (a scala.concurrent.impl.Promise$CompletionLatch)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)

        at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:242)

        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:258)

        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:187)

        at org.apache.spark.util.ThreadUtils$.awaitReady(ThreadUtils.scala:243)

        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:728)

        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)

        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)

        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2114)

        at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78)

        at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsNewAPIHadoopDataset$1(PairRDDFunctions.scala:1083)

        at org.apache.spark.rdd.PairRDDFunctions$$Lambda$787/751460639.apply$mcV$sp(Unknown Source)

        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)

        at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1081)

        at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsNewAPIHadoopFile$2(PairRDDFunctions.scala:1000)

        at org.apache.spark.rdd.PairRDDFunctions$$Lambda$786/1653309853.apply$mcV$sp(Unknown Source)

        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)

        at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:991)

        at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsNewAPIHadoopFile$1(PairRDDFunctions.scala:979)

        at org.apache.spark.rdd.PairRDDFunctions$$Lambda$785/500096147.apply$mcV$sp(Unknown Source)

        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)

        at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:979)

        at com.github.ehiggs.spark.terasort.TeraSort$.main(TeraSort.scala:63)

        at com.github.ehiggs.spark.terasort.TeraSort.main(TeraSort.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)

        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)

        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)

        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)

        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)



"VM Thread" os_prio=0 tid=0x00007fe6fc0c6000 nid=0x32e runnable



"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fe6fc029000 nid=0x308 runnable



"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fe6fc02a800 nid=0x309 runnable



"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fe6fc02c800 nid=0x30a runnable



"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fe6fc02e000 nid=0x30b runnable



"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007fe6fc030000 nid=0x30c runnable



"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007fe6fc031800 nid=0x30d runnable



"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007fe6fc033800 nid=0x30e runnable



"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007fe6fc035000 nid=0x30f runnable



"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007fe6fc037000 nid=0x310 runnable



"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007fe6fc038800 nid=0x311 runnable



"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007fe6fc03a800 nid=0x312 runnable



"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007fe6fc03c000 nid=0x313 runnable



"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007fe6fc03e000 nid=0x314 runnable



"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007fe6fc03f800 nid=0x315 runnable



"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007fe6fc041800 nid=0x316 runnable



"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007fe6fc043000 nid=0x317 runnable



"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007fe6fc045000 nid=0x318 runnable



"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007fe6fc046800 nid=0x319 runnable



"GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007fe6fc048800 nid=0x31a runnable



"GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007fe6fc04a000 nid=0x31b runnable



"GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007fe6fc04c000 nid=0x31c runnable



"GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007fe6fc04d800 nid=0x31d runnable



"GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007fe6fc04f800 nid=0x31e runnable



"GC task thread#23 (ParallelGC)" os_prio=0 tid=0x00007fe6fc051000 nid=0x31f runnable



"GC task thread#24 (ParallelGC)" os_prio=0 tid=0x00007fe6fc053000 nid=0x320 runnable



"GC task thread#25 (ParallelGC)" os_prio=0 tid=0x00007fe6fc054800 nid=0x321 runnable



"GC task thread#26 (ParallelGC)" os_prio=0 tid=0x00007fe6fc056800 nid=0x322 runnable



"GC task thread#27 (ParallelGC)" os_prio=0 tid=0x00007fe6fc058000 nid=0x323 runnable



"GC task thread#28 (ParallelGC)" os_prio=0 tid=0x00007fe6fc05a000 nid=0x324 runnable



"GC task thread#29 (ParallelGC)" os_prio=0 tid=0x00007fe6fc05b800 nid=0x325 runnable



"GC task thread#30 (ParallelGC)" os_prio=0 tid=0x00007fe6fc05d800 nid=0x326 runnable



"GC task thread#31 (ParallelGC)" os_prio=0 tid=0x00007fe6fc05f000 nid=0x327 runnable



"GC task thread#32 (ParallelGC)" os_prio=0 tid=0x00007fe6fc061000 nid=0x328 runnable



"GC task thread#33 (ParallelGC)" os_prio=0 tid=0x00007fe6fc062800 nid=0x329 runnable



"GC task thread#34 (ParallelGC)" os_prio=0 tid=0x00007fe6fc064800 nid=0x32a runnable



"GC task thread#35 (ParallelGC)" os_prio=0 tid=0x00007fe6fc066000 nid=0x32b runnable



"GC task thread#36 (ParallelGC)" os_prio=0 tid=0x00007fe6fc068000 nid=0x32c runnable



"GC task thread#37 (ParallelGC)" os_prio=0 tid=0x00007fe6fc069800 nid=0x32d runnable



"VM Periodic Task Thread" os_prio=0 tid=0x00007fe6fc131000 nid=0x342 waiting on condition



JNI global references: 1851





Regards,



           David





________________________________
From: Jonas Pfefferle <pepperjo@japf.ch>
Sent: Friday, June 28, 2019 12:54:27 AM
To: dev@crail.apache.org; David Crespi
Subject: Re: Setting up storage class 1 and 2

Hi David,


At the moment, it is possible to add a NVMf datanode even if only the RDMA
storage type is specified in the config. As you have seen this will go wrong
as soon as a client tries to connect to the datanode. Make sure to start the
RDMA datanode with the appropriate classname, see:
https://incubator-crail.readthedocs.io/en/latest/run.html
The correct classname is org.apache.crail.storage.rdma.RdmaStorageTier.

Regards,
Jonas

  On Thu, 27 Jun 2019 23:09:26 +0000
  David Crespi <david.crespi@storedgesystems.com> wrote:
> Hi,
> I’m trying to integrate the storage classes and I’m hitting another
>issue when running terasort and just
> using the crail-shuffle with HDFS as the tmp storage.  The program
>just sits, after the following
> message:
> 19/06/27 15:59:20 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser: closed
> 19/06/27 15:59:20 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser: stopped, remaining
>connections 0
>
> During this run, I’ve removed the two crail nvmf (class 1 and 2)
>containers from the server, and I’m only running
> the namenode and a rdma storage class 1 datanode.  My spark
>configuration is also now only looking at
> the rdma class.  It looks as though it’s picking up the NVMf IP and
>port in the INFO messages seen below.
> I must be configuring something wrong, but I’ve not been able to
>track it down.  Any thoughts?
>
>
> ************************************
>         TeraSort
> ************************************
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
>[jar:file:/crail/jars/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>[jar:file:/crail/jars/jnvmf-1.6-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>[jar:file:/crail/jars/disni-2.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
>[jar:file:/usr/spark-2.4.2/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 19/06/27 15:59:07 WARN NativeCodeLoader: Unable to load
>native-hadoop library for your platform... using builtin-java classes
>where applicable
> 19/06/27 15:59:07 INFO SparkContext: Running Spark version 2.4.2
> 19/06/27 15:59:07 INFO SparkContext: Submitted application: TeraSort
> 19/06/27 15:59:07 INFO SecurityManager: Changing view acls to:
>hduser
> 19/06/27 15:59:07 INFO SecurityManager: Changing modify acls to:
>hduser
> 19/06/27 15:59:07 INFO SecurityManager: Changing view acls groups
>to:
> 19/06/27 15:59:07 INFO SecurityManager: Changing modify acls groups
>to:
> 19/06/27 15:59:07 INFO SecurityManager: SecurityManager:
>authentication disabled; ui acls disabled; users  with view
>permissions: Set(hduser); groups with view permissions: Set(); users
> with modify permissions: Set(hduser); groups with modify
>permissions: Set()
> 19/06/27 15:59:08 DEBUG InternalLoggerFactory: Using SLF4J as the
>default logging framework
> 19/06/27 15:59:08 DEBUG InternalThreadLocalMap:
>-Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
> 19/06/27 15:59:08 DEBUG InternalThreadLocalMap:
>-Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
> 19/06/27 15:59:08 DEBUG MultithreadEventLoopGroup:
>-Dio.netty.eventLoopThreads: 112
> 19/06/27 15:59:08 DEBUG PlatformDependent0: -Dio.netty.noUnsafe:
>false
> 19/06/27 15:59:08 DEBUG PlatformDependent0: Java version: 8
> 19/06/27 15:59:08 DEBUG PlatformDependent0:
>sun.misc.Unsafe.theUnsafe: available
> 19/06/27 15:59:08 DEBUG PlatformDependent0:
>sun.misc.Unsafe.copyMemory: available
> 19/06/27 15:59:08 DEBUG PlatformDependent0: java.nio.Buffer.address:
>available
> 19/06/27 15:59:08 DEBUG PlatformDependent0: direct buffer
>constructor: available
> 19/06/27 15:59:08 DEBUG PlatformDependent0: java.nio.Bits.unaligned:
>available, true
> 19/06/27 15:59:08 DEBUG PlatformDependent0:
>jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable
>prior to Java9
> 19/06/27 15:59:08 DEBUG PlatformDependent0:
>java.nio.DirectByteBuffer.<init>(long, int): available
> 19/06/27 15:59:08 DEBUG PlatformDependent: sun.misc.Unsafe:
>available
> 19/06/27 15:59:08 DEBUG PlatformDependent: -Dio.netty.tmpdir: /tmp
>(java.io.tmpdir)
> 19/06/27 15:59:08 DEBUG PlatformDependent: -Dio.netty.bitMode: 64
>(sun.arch.data.model)
> 19/06/27 15:59:08 DEBUG PlatformDependent:
>-Dio.netty.noPreferDirect: false
> 19/06/27 15:59:08 DEBUG PlatformDependent:
>-Dio.netty.maxDirectMemory: 1029177344 bytes
> 19/06/27 15:59:08 DEBUG PlatformDependent:
>-Dio.netty.uninitializedArrayAllocationThreshold: -1
> 19/06/27 15:59:08 DEBUG CleanerJava6: java.nio.ByteBuffer.cleaner():
>available
> 19/06/27 15:59:08 DEBUG NioEventLoop:
>-Dio.netty.noKeySetOptimization: false
> 19/06/27 15:59:08 DEBUG NioEventLoop:
>-Dio.netty.selectorAutoRebuildThreshold: 512
> 19/06/27 15:59:08 DEBUG PlatformDependent:
>org.jctools-core.MpscChunkedArrayQueue: available
> 19/06/27 15:59:08 DEBUG ResourceLeakDetector:
>-Dio.netty.leakDetection.level: simple
> 19/06/27 15:59:08 DEBUG ResourceLeakDetector:
>-Dio.netty.leakDetection.targetRecords: 4
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.numHeapArenas: 9
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.numDirectArenas: 10
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.pageSize: 8192
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.maxOrder: 11
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.chunkSize: 16777216
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.tinyCacheSize: 512
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.smallCacheSize: 256
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.normalCacheSize: 64
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.maxCachedBufferCapacity: 32768
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.cacheTrimInterval: 8192
> 19/06/27 15:59:08 DEBUG PooledByteBufAllocator:
>-Dio.netty.allocator.useCacheForAllThreads: true
> 19/06/27 15:59:08 DEBUG DefaultChannelId: -Dio.netty.processId: 2236
>(auto-detected)
> 19/06/27 15:59:08 DEBUG NetUtil: -Djava.net.preferIPv4Stack: false
> 19/06/27 15:59:08 DEBUG NetUtil: -Djava.net.preferIPv6Addresses:
>false
> 19/06/27 15:59:08 DEBUG NetUtil: Loopback interface: lo (lo,
>127.0.0.1)
> 19/06/27 15:59:08 DEBUG NetUtil: /proc/sys/net/core/somaxconn: 128
> 19/06/27 15:59:08 DEBUG DefaultChannelId: -Dio.netty.machineId:
>02:42:ac:ff:fe:1b:00:02 (auto-detected)
> 19/06/27 15:59:08 DEBUG ByteBufUtil: -Dio.netty.allocator.type:
>pooled
> 19/06/27 15:59:08 DEBUG ByteBufUtil:
>-Dio.netty.threadLocalDirectBufferSize: 65536
> 19/06/27 15:59:08 DEBUG ByteBufUtil:
>-Dio.netty.maxThreadLocalCharBufferSize: 16384
> 19/06/27 15:59:08 DEBUG TransportServer: Shuffle server started on
>port: 36915
> 19/06/27 15:59:08 INFO Utils: Successfully started service
>'sparkDriver' on port 36915.
> 19/06/27 15:59:08 DEBUG SparkEnv: Using serializer: class
>org.apache.spark.serializer.KryoSerializer
> 19/06/27 15:59:08 INFO SparkEnv: Registering MapOutputTracker
> 19/06/27 15:59:08 DEBUG MapOutputTrackerMasterEndpoint: init
> 19/06/27 15:59:08 INFO CrailShuffleManager: crail shuffle started
> 19/06/27 15:59:08 INFO SparkEnv: Registering BlockManagerMaster
> 19/06/27 15:59:08 INFO BlockManagerMasterEndpoint: Using
>org.apache.spark.storage.DefaultTopologyMapper for getting topology
>information
> 19/06/27 15:59:08 INFO BlockManagerMasterEndpoint:
>BlockManagerMasterEndpoint up
> 19/06/27 15:59:08 INFO DiskBlockManager: Created local directory at
>/tmp/blockmgr-15237510-f459-40e3-8390-10f4742930a5
> 19/06/27 15:59:08 DEBUG DiskBlockManager: Adding shutdown hook
> 19/06/27 15:59:08 INFO MemoryStore: MemoryStore started with
>capacity 366.3 MB
> 19/06/27 15:59:08 INFO SparkEnv: Registering OutputCommitCoordinator
> 19/06/27 15:59:08 DEBUG
>OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: init
> 19/06/27 15:59:08 DEBUG SecurityManager: Created SSL options for ui:
>SSLOptions{enabled=false, port=None, keyStore=None,
>keyStorePassword=None, trustStore=None, trustStorePassword=None,
>protocol=None, enabledAlgorithms=Set()}
> 19/06/27 15:59:08 INFO Utils: Successfully started service 'SparkUI'
>on port 4040.
> 19/06/27 15:59:08 INFO SparkUI: Bound SparkUI to 0.0.0.0, and
>started at http://192.168.1.161:4040
> 19/06/27 15:59:08 INFO SparkContext: Added JAR
>file:/spark-terasort/target/spark-terasort-1.1-SNAPSHOT-jar-with-dependencies.jar
>at
>spark://master:36915/jars/spark-terasort-1.1-SNAPSHOT-jar-with-dependencies.jar
>with timestamp 1561676348562
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint:
>Connecting to master spark://master:7077...
> 19/06/27 15:59:08 DEBUG TransportClientFactory: Creating new
>connection to master/192.168.3.13:7077
> 19/06/27 15:59:08 DEBUG AbstractByteBuf:
>-Dio.netty.buffer.bytebuf.checkAccessible: true
> 19/06/27 15:59:08 DEBUG ResourceLeakDetectorFactory: Loaded default
>ResourceLeakDetector: io.netty.util.ResourceLeakDetector@5b1bb5d2
> 19/06/27 15:59:08 DEBUG TransportClientFactory: Connection to
>master/192.168.3.13:7077 successful, running bootstraps...
> 19/06/27 15:59:08 INFO TransportClientFactory: Successfully created
>connection to master/192.168.3.13:7077 after 41 ms (0 ms spent in
>bootstraps)
> 19/06/27 15:59:08 DEBUG Recycler:
>-Dio.netty.recycler.maxCapacityPerThread: 32768
> 19/06/27 15:59:08 DEBUG Recycler:
>-Dio.netty.recycler.maxSharedCapacityFactor: 2
> 19/06/27 15:59:08 DEBUG Recycler: -Dio.netty.recycler.linkCapacity:
>16
> 19/06/27 15:59:08 DEBUG Recycler: -Dio.netty.recycler.ratio: 8
> 19/06/27 15:59:08 INFO StandaloneSchedulerBackend: Connected to
>Spark cluster with app ID app-20190627155908-0005
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>added: app-20190627155908-0005/0 on
>worker-20190627152154-192.168.3.11-8882 (192.168.3.11:8882) with 2
>core(s)
> 19/06/27 15:59:08 INFO StandaloneSchedulerBackend: Granted executor
>ID app-20190627155908-0005/0 on hostPort 192.168.3.11:8882 with 2
>core(s), 1024.0 MB RAM
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>added: app-20190627155908-0005/1 on
>worker-20190627152150-192.168.3.12-8881 (192.168.3.12:8881) with 2
>core(s)
> 19/06/27 15:59:08 INFO StandaloneSchedulerBackend: Granted executor
>ID app-20190627155908-0005/1 on hostPort 192.168.3.12:8881 with 2
>core(s), 1024.0 MB RAM
> 19/06/27 15:59:08 DEBUG TransportServer: Shuffle server started on
>port: 39189
> 19/06/27 15:59:08 INFO Utils: Successfully started service
>'org.apache.spark.network.netty.NettyBlockTransferService' on port
>39189.
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>added: app-20190627155908-0005/2 on
>worker-20190627152203-192.168.3.9-8884 (192.168.3.9:8884) with 2
>core(s)
> 19/06/27 15:59:08 INFO NettyBlockTransferService: Server created on
>master:39189
> 19/06/27 15:59:08 INFO StandaloneSchedulerBackend: Granted executor
>ID app-20190627155908-0005/2 on hostPort 192.168.3.9:8884 with 2
>core(s), 1024.0 MB RAM
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>added: app-20190627155908-0005/3 on
>worker-20190627152158-192.168.3.10-8883 (192.168.3.10:8883) with 2
>core(s)
> 19/06/27 15:59:08 INFO StandaloneSchedulerBackend: Granted executor
>ID app-20190627155908-0005/3 on hostPort 192.168.3.10:8883 with 2
>core(s), 1024.0 MB RAM
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>added: app-20190627155908-0005/4 on
>worker-20190627152207-192.168.3.8-8885 (192.168.3.8:8885) with 2
>core(s)
> 19/06/27 15:59:08 INFO BlockManager: Using
>org.apache.spark.storage.RandomBlockReplicationPolicy for block
>replication policy
> 19/06/27 15:59:08 INFO StandaloneSchedulerBackend: Granted executor
>ID app-20190627155908-0005/4 on hostPort 192.168.3.8:8885 with 2
>core(s), 1024.0 MB RAM
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>updated: app-20190627155908-0005/0 is now RUNNING
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>updated: app-20190627155908-0005/3 is now RUNNING
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>updated: app-20190627155908-0005/4 is now RUNNING
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>updated: app-20190627155908-0005/1 is now RUNNING
> 19/06/27 15:59:08 INFO StandaloneAppClient$ClientEndpoint: Executor
>updated: app-20190627155908-0005/2 is now RUNNING
> 19/06/27 15:59:08 INFO BlockManagerMaster: Registering BlockManager
>BlockManagerId(driver, master, 39189, None)
> 19/06/27 15:59:08 DEBUG DefaultTopologyMapper: Got a request for
>master
> 19/06/27 15:59:08 INFO BlockManagerMasterEndpoint: Registering block
>manager master:39189 with 366.3 MB RAM, BlockManagerId(driver,
>master, 39189, None)
> 19/06/27 15:59:08 INFO BlockManagerMaster: Registered BlockManager
>BlockManagerId(driver, master, 39189, None)
> 19/06/27 15:59:08 INFO BlockManager: Initialized BlockManager:
>BlockManagerId(driver, master, 39189, None)
> 19/06/27 15:59:09 INFO StandaloneSchedulerBackend: SchedulerBackend
>is ready for scheduling beginning after reached
>minRegisteredResourcesRatio: 0.0
> 19/06/27 15:59:09 DEBUG SparkContext: Adding shutdown hook
> 19/06/27 15:59:09 DEBUG BlockReaderLocal:
>dfs.client.use.legacy.blockreader.local = false
> 19/06/27 15:59:09 DEBUG BlockReaderLocal:
>dfs.client.read.shortcircuit = false
> 19/06/27 15:59:09 DEBUG BlockReaderLocal:
>dfs.client.domain.socket.data.traffic = false
> 19/06/27 15:59:09 DEBUG BlockReaderLocal: dfs.domain.socket.path =
> 19/06/27 15:59:09 DEBUG RetryUtils: multipleLinearRandomRetry = null
> 19/06/27 15:59:09 DEBUG Server: rpcKind=RPC_PROTOCOL_BUFFER,
>rpcRequestWrapperClass=class
>org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper,
>rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@23f3dbf0
> 19/06/27 15:59:09 DEBUG Client: getting client out of cache:
>org.apache.hadoop.ipc.Client@3ed03652
> 19/06/27 15:59:09 DEBUG PerformanceAdvisory: Both short-circuit
>local reads and UNIX domain socket are disabled.
> 19/06/27 15:59:09 DEBUG DataTransferSaslUtil: DataTransferProtocol
>not using SaslPropertiesResolver, no QOP found in configuration for
>dfs.data.transfer.protection
> 19/06/27 15:59:10 INFO MemoryStore: Block broadcast_0 stored as
>values in memory (estimated size 288.9 KB, free 366.0 MB)
> 19/06/27 15:59:10 DEBUG BlockManager: Put block broadcast_0 locally
>took  115 ms
> 19/06/27 15:59:10 DEBUG BlockManager: Putting block broadcast_0
>without replication took  117 ms
> 19/06/27 15:59:10 INFO MemoryStore: Block broadcast_0_piece0 stored
>as bytes in memory (estimated size 23.8 KB, free 366.0 MB)
> 19/06/27 15:59:10 INFO BlockManagerInfo: Added broadcast_0_piece0 in
>memory on master:39189 (size: 23.8 KB, free: 366.3 MB)
> 19/06/27 15:59:10 DEBUG BlockManagerMaster: Updated info of block
>broadcast_0_piece0
> 19/06/27 15:59:10 DEBUG BlockManager: Told master about block
>broadcast_0_piece0
> 19/06/27 15:59:10 DEBUG BlockManager: Put block broadcast_0_piece0
>locally took  6 ms
> 19/06/27 15:59:10 DEBUG BlockManager: Putting block
>broadcast_0_piece0 without replication took  6 ms
> 19/06/27 15:59:10 INFO SparkContext: Created broadcast 0 from
>newAPIHadoopFile at TeraSort.scala:60
> 19/06/27 15:59:10 DEBUG Client: The ping interval is 60000 ms.
> 19/06/27 15:59:10 DEBUG Client: Connecting to
>NameNode-1/192.168.3.7:54310
> 19/06/27 15:59:10 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser: starting, having
>connections 1
> 19/06/27 15:59:10 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser sending #0
> 19/06/27 15:59:10 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser got value #0
> 19/06/27 15:59:10 DEBUG ProtobufRpcEngine: Call: getFileInfo took
>31ms
> 19/06/27 15:59:10 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser sending #1
> 19/06/27 15:59:10 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser got value #1
> 19/06/27 15:59:10 DEBUG ProtobufRpcEngine: Call: getListing took 5ms
> 19/06/27 15:59:10 DEBUG FileInputFormat: Time taken to get
>FileStatuses: 134
> 19/06/27 15:59:10 INFO FileInputFormat: Total input paths to process
>: 2
> 19/06/27 15:59:10 DEBUG FileInputFormat: Total # of splits generated
>by getSplits: 2, TimeTaken: 139
> 19/06/27 15:59:10 DEBUG FileCommitProtocol: Creating committer
>org.apache.spark.internal.io.HadoopMapReduceCommitProtocol; job 1;
>output=hdfs://NameNode-1:54310/tmp/data_sort; dynamic=false
> 19/06/27 15:59:10 DEBUG FileCommitProtocol: Using (String, String,
>Boolean) constructor
> 19/06/27 15:59:10 INFO FileOutputCommitter: File Output Committer
>Algorithm version is 1
> 19/06/27 15:59:10 DEBUG DFSClient: /tmp/data_sort/_temporary/0:
>masked=rwxr-xr-x
> 19/06/27 15:59:10 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser sending #2
> 19/06/27 15:59:10 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser got value #2
> 19/06/27 15:59:10 DEBUG ProtobufRpcEngine: Call: mkdirs took 3ms
> 19/06/27 15:59:10 DEBUG ClosureCleaner: Cleaning lambda:
>$anonfun$write$1
> 19/06/27 15:59:10 DEBUG ClosureCleaner:  +++ Lambda closure
>($anonfun$write$1) is now cleaned +++
> 19/06/27 15:59:10 INFO SparkContext: Starting job: runJob at
>SparkHadoopWriter.scala:78
> 19/06/27 15:59:10 INFO CrailDispatcher: CrailStore starting version
>400
> 19/06/27 15:59:10 INFO CrailDispatcher: spark.crail.deleteonclose
>false
> 19/06/27 15:59:10 INFO CrailDispatcher: spark.crail.deleteOnStart
>true
> 19/06/27 15:59:10 INFO CrailDispatcher: spark.crail.preallocate 0
> 19/06/27 15:59:10 INFO CrailDispatcher: spark.crail.writeAhead 0
> 19/06/27 15:59:10 INFO CrailDispatcher: spark.crail.debug false
> 19/06/27 15:59:10 INFO CrailDispatcher: spark.crail.serializer
>org.apache.spark.serializer.CrailSparkSerializer
> 19/06/27 15:59:10 INFO CrailDispatcher: spark.crail.shuffle.affinity
>true
> 19/06/27 15:59:10 INFO CrailDispatcher:
>spark.crail.shuffle.outstanding 1
> 19/06/27 15:59:10 INFO CrailDispatcher:
>spark.crail.shuffle.storageclass 0
> 19/06/27 15:59:10 INFO CrailDispatcher:
>spark.crail.broadcast.storageclass 0
> 19/06/27 15:59:10 INFO crail: creating singleton crail file system
> 19/06/27 15:59:10 INFO crail: crail.version 3101
> 19/06/27 15:59:10 INFO crail: crail.directorydepth 16
> 19/06/27 15:59:10 INFO crail: crail.tokenexpiration 10
> 19/06/27 15:59:10 INFO crail: crail.blocksize 1048576
> 19/06/27 15:59:10 INFO crail: crail.cachelimit 0
> 19/06/27 15:59:10 INFO crail: crail.cachepath /dev/hugepages/cache
> 19/06/27 15:59:10 INFO crail: crail.user crail
> 19/06/27 15:59:10 INFO crail: crail.shadowreplication 1
> 19/06/27 15:59:10 INFO crail: crail.debug true
> 19/06/27 15:59:10 INFO crail: crail.statistics true
> 19/06/27 15:59:10 INFO crail: crail.rpctimeout 1000
> 19/06/27 15:59:10 INFO crail: crail.datatimeout 1000
> 19/06/27 15:59:10 INFO crail: crail.buffersize 1048576
> 19/06/27 15:59:10 INFO crail: crail.slicesize 65536
> 19/06/27 15:59:10 INFO crail: crail.singleton true
> 19/06/27 15:59:10 INFO crail: crail.regionsize 1073741824
> 19/06/27 15:59:10 INFO crail: crail.directoryrecord 512
> 19/06/27 15:59:10 INFO crail: crail.directoryrandomize true
> 19/06/27 15:59:10 INFO crail: crail.cacheimpl
>org.apache.crail.memory.MappedBufferCache
> 19/06/27 15:59:10 INFO crail: crail.locationmap
> 19/06/27 15:59:10 INFO crail: crail.namenode.address
>crail://192.168.1.164:9060
> 19/06/27 15:59:10 INFO crail: crail.namenode.blockselection
>roundrobin
> 19/06/27 15:59:10 INFO crail: crail.namenode.fileblocks 16
> 19/06/27 15:59:10 INFO crail: crail.namenode.rpctype
>org.apache.crail.namenode.rpc.tcp.TcpNameNode
> 19/06/27 15:59:10 INFO crail: crail.namenode.log
> 19/06/27 15:59:10 INFO crail: crail.storage.types
>org.apache.crail.storage.rdma.RdmaStorageTier
> 19/06/27 15:59:10 INFO crail: crail.storage.classes 1
> 19/06/27 15:59:10 INFO crail: crail.storage.rootclass 0
> 19/06/27 15:59:10 INFO crail: crail.storage.keepalive 2
> 19/06/27 15:59:10 INFO crail: buffer cache, allocationCount 0,
>bufferCount 1024
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.interface eth0
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.port 50020
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.storagelimit
>4294967296
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.allocationsize
>1073741824
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.datapath
>/dev/hugepages/rdma
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.localmap true
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.queuesize 32
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.type passive
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.backlog 100
> 19/06/27 15:59:10 INFO crail: crail.storage.rdma.connecttimeout 1000
> 19/06/27 15:59:10 INFO narpc: new NaRPC server group v1.0,
>queueDepth 32, messageSize 512, nodealy true
> 19/06/27 15:59:10 INFO crail: crail.namenode.tcp.queueDepth 32
> 19/06/27 15:59:10 INFO crail: crail.namenode.tcp.messageSize 512
> 19/06/27 15:59:10 INFO crail: crail.namenode.tcp.cores 1
> 19/06/27 15:59:10 INFO crail: connected to namenode(s)
>/192.168.1.164:9060
> 19/06/27 15:59:10 INFO CrailDispatcher: creating main dir /spark
> 19/06/27 15:59:10 INFO crail: lookupDirectory: path /spark
> 19/06/27 15:59:10 INFO CrailDispatcher: creating main dir /spark
> 19/06/27 15:59:10 INFO crail: createNode: name /spark, type
>DIRECTORY, storageAffinity 0, locationAffinity 0
> 19/06/27 15:59:10 INFO crail: CoreOutputStream, open, path /, fd 0,
>streamId 1, isDir true, writeHint 0
> 19/06/27 15:59:10 INFO crail: passive data client
> 19/06/27 15:59:10 INFO disni: creating  RdmaProvider of type 'nat'
> 19/06/27 15:59:10 INFO disni: jverbs jni version 32
> 19/06/27 15:59:10 INFO disni: sock_addr_in size mismatch, jverbs
>size 28, native size 16
> 19/06/27 15:59:10 INFO disni: IbvRecvWR size match, jverbs size 32,
>native size 32
> 19/06/27 15:59:10 INFO disni: IbvSendWR size mismatch, jverbs size
>72, native size 128
> 19/06/27 15:59:10 INFO disni: IbvWC size match, jverbs size 48,
>native size 48
> 19/06/27 15:59:10 INFO disni: IbvSge size match, jverbs size 16,
>native size 16
> 19/06/27 15:59:10 INFO disni: Remote addr offset match, jverbs size
>40, native size 40
> 19/06/27 15:59:10 INFO disni: Rkey offset match, jverbs size 48,
>native size 48
> 19/06/27 15:59:10 INFO disni: createEventChannel, objId
>139811924587312
> 19/06/27 15:59:10 INFO disni: passive endpoint group, maxWR 32,
>maxSge 4, cqSize 64
> 19/06/27 15:59:10 INFO disni: launching cm processor, cmChannel 0
> 19/06/27 15:59:10 INFO disni: createId, id 139811924676432
> 19/06/27 15:59:10 INFO disni: new client endpoint, id 0, idPriv 0
> 19/06/27 15:59:10 INFO disni: resolveAddr, addres
>/192.168.3.100:4420
> 19/06/27 15:59:10 INFO disni: resolveRoute, id 0
> 19/06/27 15:59:10 INFO disni: allocPd, objId 139811924679808
> 19/06/27 15:59:10 INFO disni: setting up protection domain, context
>467, pd 1
> 19/06/27 15:59:10 INFO disni: setting up cq processor
> 19/06/27 15:59:10 INFO disni: new endpoint CQ processor
> 19/06/27 15:59:10 INFO disni: createCompChannel, context
>139810647883744
> 19/06/27 15:59:10 INFO disni: createCQ, objId 139811924680688, ncqe
>64
> 19/06/27 15:59:10 INFO disni: createQP, objId 139811924691192,
>send_wr size 32, recv_wr_size 32
> 19/06/27 15:59:10 INFO disni: connect, id 0
> 19/06/27 15:59:10 INFO disni: got event type + UNKNOWN, srcAddress
>/192.168.3.13:43273, dstAddress /192.168.3.100:4420
> 19/06/27 15:59:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:
>Registered executor NettyRpcEndpointRef(spark-client://Executor)
>(192.168.3.11:35854) with ID 0
> 19/06/27 15:59:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:
>Registered executor NettyRpcEndpointRef(spark-client://Executor)
>(192.168.3.12:44312) with ID 1
> 19/06/27 15:59:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:
>Registered executor NettyRpcEndpointRef(spark-client://Executor)
>(192.168.3.8:34774) with ID 4
> 19/06/27 15:59:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:
>Registered executor NettyRpcEndpointRef(spark-client://Executor)
>(192.168.3.9:58808) with ID 2
> 19/06/27 15:59:11 DEBUG DefaultTopologyMapper: Got a request for
>192.168.3.11
> 19/06/27 15:59:11 INFO BlockManagerMasterEndpoint: Registering block
>manager 192.168.3.11:41919 with 366.3 MB RAM, BlockManagerId(0,
>192.168.3.11, 41919, None)
> 19/06/27 15:59:11 DEBUG DefaultTopologyMapper: Got a request for
>192.168.3.12
> 19/06/27 15:59:11 INFO BlockManagerMasterEndpoint: Registering block
>manager 192.168.3.12:46697 with 366.3 MB RAM, BlockManagerId(1,
>192.168.3.12, 46697, None)
> 19/06/27 15:59:11 DEBUG DefaultTopologyMapper: Got a request for
>192.168.3.8
> 19/06/27 15:59:11 INFO BlockManagerMasterEndpoint: Registering block
>manager 192.168.3.8:37281 with 366.3 MB RAM, BlockManagerId(4,
>192.168.3.8, 37281, None)
> 19/06/27 15:59:11 DEBUG DefaultTopologyMapper: Got a request for
>192.168.3.9
> 19/06/27 15:59:11 INFO BlockManagerMasterEndpoint: Registering block
>manager 192.168.3.9:43857 with 366.3 MB RAM, BlockManagerId(2,
>192.168.3.9, 43857, None)
> 19/06/27 15:59:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:
>Registered executor NettyRpcEndpointRef(spark-client://Executor)
>(192.168.3.10:40100) with ID 3
> 19/06/27 15:59:11 DEBUG DefaultTopologyMapper: Got a request for
>192.168.3.10
> 19/06/27 15:59:11 INFO BlockManagerMasterEndpoint: Registering block
>manager 192.168.3.10:38527 with 366.3 MB RAM, BlockManagerId(3,
>192.168.3.10, 38527, None)
> 19/06/27 15:59:20 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser: closed
> 19/06/27 15:59:20 DEBUG Client: IPC Client (1998371610) connection
>to NameNode-1/192.168.3.7:54310 from hduser: stopped, remaining
>connections 0
>
>
> Regards,
>
>           David
>



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message