spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kanagha Pradha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-18343) FileSystem$Statistics$StatisticsDataReferenceCleaner hangs on s3 write
Date Thu, 22 Jun 2017 23:44:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060203#comment-16060203 ] 

Kanagha Pradha commented on SPARK-18343:
----------------------------------------

I am getting the same error with spark 2.0.2 - scala 2.11 version. As [~lminer] mentioned, I looked at the versions.
the spark pom.xml by default references json4s - 3.2.11 version. I'm not sure what is the reason.
Also, this error occurs intermittently and several times spark-submit succeeds. Any pointers on what could be causing this?


Thread 24 (org.apache.hadoop.hdfs.PeerCache@1c80e49b):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 23
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255)
    org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46)
    org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124)
    java.lang.Thread.run(Thread.java:748)
Thread 22 (communication thread):
  State: RUNNABLE
  Blocked count: 20
  Waited count: 63
  Stack:
    sun.management.ThreadImpl.getThreadInfo1(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:168)
    org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:223)
    org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:785)
    java.lang.Thread.run(Thread.java:748)
Thread 16 (process reaper):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
    java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    java.lang.Thread.run(Thread.java:748)
Thread 14 (org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.lang.ref.ReferenceQueue$Lock@407a7f2a
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
    org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3060)
    java.lang.Thread.run(Thread.java:748)
Thread 13 (IPC Parameter Sending Thread #0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 31
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
    java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    java.lang.Thread.run(Thread.java:748)
Thread 4 (Signal Dispatcher):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
Thread 3 (Finalizer):
  State: WAITING
  Blocked count: 567
  Waited count: 6
  Waiting on java.lang.ref.ReferenceQueue$Lock@2cb2fc20
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Thread 2 (Reference Handler):
  State: WAITING
  Blocked count: 9
  Waited count: 5
  Waiting on java.lang.ref.Reference$Lock@4f4c4b1a
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:502)
    java.lang.ref.Reference.tryHandlePending(Reference.java:191)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Jun 22, 2017 11:01:23 PM org.apache.hadoop.mapred.Task run
WARNING: Last retry, killing attempt_1498171863687_0003_m_000000_0

> FileSystem$Statistics$StatisticsDataReferenceCleaner hangs on s3 write
> ----------------------------------------------------------------------
>
>                 Key: SPARK-18343
>                 URL: https://issues.apache.org/jira/browse/SPARK-18343
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.0.1
>         Environment: Spark 2.0.1
> Hadoop 2.7.1
> Mesos 1.0.1
> Ubuntu 14.04
>            Reporter: Luke Miner
>
> I have a driver program where I write read data in from Cassandra using spark, perform some operations, and then write out to JSON on S3. The program runs fine when I use Spark 1.6.1 and the spark-cassandra-connector 1.6.0-M1.
> However, if I try to upgrade to Spark 2.0.1 (hadoop 2.7.1) and spark-cassandra-connector 2.0.0-M3, the program completes in the sense that all the expected files are written to S3, but the program never terminates.
> I do run `sc.stop()` at the end of the program. I am also using Mesos 1.0.1. In both cases I use the default output committer.
> From the thread dump (included below) it seems like it could be waiting on: `org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner`
> Code snippet:
> {code}
>     // get MongoDB oplog operations
>     val operations = sc.cassandraTable[JsonOperation](keyspace, namespace)
>       .where("ts >= ? AND ts < ?", minTimestamp, maxTimestamp)
>     
>     // replay oplog operations into documents
>     val documents = operations
>       .spanBy(op => op.id)
>       .map { case (id: String, ops: Iterable[T]) => (id, apply(ops)) }
>       .filter { case (id, result) => result.isInstanceOf[Document] }
>       .map { case (id, document) => MergedDocument(id = id, document = document
>         .asInstanceOf[Document])
>       }
>     
>     // write documents to json on s3
>     documents
>       .map(document => document.toJson)
>       .coalesce(partitions)
>       .saveAsTextFile(path, classOf[GzipCodec])
>     sc.stop()
> {code}
> build.sbt
> {code}
> scalaVersion := "2.11.8"
> ivyScala := ivyScala.value map {
>   _.copy(overrideScalaVersion = true)
> }
> resolvers += Resolver.mavenLocal
> // spark
> libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" % "provided"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" % "provided"
> // spark cassandra connector
> libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.0-M3" excludeAll
>   ExclusionRule(organization = "org.apache.spark")
> // other libraries
> libraryDependencies += "org.json4s" %% "json4s-native" % "3.2.11"
> libraryDependencies += "org.rogach" %% "scallop" % "2.0.1"
> libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "1.8.0"
> // test
> libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.6" % "test"
> libraryDependencies += "junit" % "junit" % "4.11" % "test"
> libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test"
> assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
> net.virtualvoid.sbt.graph.Plugin.graphSettings
> fork := true
> assemblyShadeRules in assembly := Seq(
>         ShadeRule.rename("com.google.common.**" -> "shadedguava.@1").inAll
>       )
>       
> assemblyMergeStrategy in assembly := {
>   case PathList(ps@_*) if ps.last endsWith ".properties" => MergeStrategy.first
>   case x =>
>     val oldStrategy = (assemblyMergeStrategy in assembly).value
>     oldStrategy(x)
> }
> scalacOptions += "-deprecation"
> {code}
> Thread dump on the driver:
> {code}
>     60  context-cleaner-periodic-gc TIMED_WAITING
>     46  dag-scheduler-event-loop    WAITING
>     4389    DestroyJavaVM   RUNNABLE
>     12  dispatcher-event-loop-0 WAITING
>     13  dispatcher-event-loop-1 WAITING
>     14  dispatcher-event-loop-2 WAITING
>     15  dispatcher-event-loop-3 WAITING
>     47  driver-revive-thread    TIMED_WAITING
>     3   Finalizer   WAITING
>     82  ForkJoinPool-1-worker-17    WAITING
>     43  heartbeat-receiver-event-loop-thread    TIMED_WAITING
>     93  java-sdk-http-connection-reaper TIMED_WAITING
>     4387    java-sdk-progress-listener-callback-thread  WAITING
>     25  map-output-dispatcher-0 WAITING
>     26  map-output-dispatcher-1 WAITING
>     27  map-output-dispatcher-2 WAITING
>     28  map-output-dispatcher-3 WAITING
>     29  map-output-dispatcher-4 WAITING
>     30  map-output-dispatcher-5 WAITING
>     31  map-output-dispatcher-6 WAITING
>     32  map-output-dispatcher-7 WAITING
>     48  MesosCoarseGrainedSchedulerBackend-mesos-driver RUNNABLE
>     44  netty-rpc-env-timeout   TIMED_WAITING
>     92  org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner   WAITING
>     62  pool-19-thread-1    TIMED_WAITING
>     2   Reference Handler   WAITING
>     61  Scheduler-1112394071    TIMED_WAITING
>     20  shuffle-server-0    RUNNABLE
>     55  shuffle-server-0    RUNNABLE
>     21  shuffle-server-1    RUNNABLE
>     56  shuffle-server-1    RUNNABLE
>     22  shuffle-server-2    RUNNABLE
>     57  shuffle-server-2    RUNNABLE
>     23  shuffle-server-3    RUNNABLE
>     58  shuffle-server-3    RUNNABLE
>     4   Signal Dispatcher   RUNNABLE
>     59  Spark Context Cleaner   TIMED_WAITING
>     9   SparkListenerBus    WAITING
>     35  SparkUI-35-selector-ServerConnectorManager@651d3734/0   RUNNABLE
>     36  SparkUI-36-acceptor-0@467924cb-ServerConnector@3b5eaf92{HTTP/1.1}{0.0.0.0:4040} RUNNABLE
>     37  SparkUI-37-selector-ServerConnectorManager@651d3734/1   RUNNABLE
>     38  SparkUI-38  TIMED_WAITING
>     39  SparkUI-39  TIMED_WAITING
>     40  SparkUI-40  TIMED_WAITING
>     41  SparkUI-41  RUNNABLE
>     42  SparkUI-42  TIMED_WAITING
>     438 task-result-getter-0    WAITING
>     450 task-result-getter-1    WAITING
>     489 task-result-getter-2    WAITING
>     492 task-result-getter-3    WAITING
>     75  threadDeathWatcher-2-1  TIMED_WAITING
>     45  Timer-0 WAITING
> {code}
> Thread dump on the executors. It's the same on all of them:
> {code}
>     24  dispatcher-event-loop-0 WAITING
>     25  dispatcher-event-loop-1 WAITING
>     26  dispatcher-event-loop-2 RUNNABLE
>     27  dispatcher-event-loop-3 WAITING
>     39  driver-heartbeater  TIMED_WAITING
>     3   Finalizer   WAITING
>     58  java-sdk-http-connection-reaper TIMED_WAITING
>     75  java-sdk-progress-listener-callback-thread  WAITING
>     1   main    TIMED_WAITING
>     33  netty-rpc-env-timeout   TIMED_WAITING
>     55  org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner   WAITING
>     59  pool-17-thread-1    TIMED_WAITING
>     2   Reference Handler   WAITING
>     28  shuffle-client-0    RUNNABLE
>     35  shuffle-client-0    RUNNABLE
>     41  shuffle-client-0    RUNNABLE
>     37  shuffle-server-0    RUNNABLE
>     5   Signal Dispatcher   RUNNABLE
>     23  threadDeathWatcher-2-1  TIMED_WAITING
> {code}
> Jstack of an executor:
> {code}
> ubuntu@ip-10-0-230-88:~$ sudo jstack  21811
> 2016-11-08 21:38:02
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode):
> "Attach Listener" daemon prio=10 tid=0x00007f8234003800 nid=0x5a4c waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "java-sdk-progress-listener-callback-thread" daemon prio=10 tid=0x00007f8218001000 nid=0x55c5 waiting on condition [0x00007f81e98d5000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000078797f4f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "pool-17-thread-1" daemon prio=10 tid=0x00007f82141f9000 nid=0x5597 waiting on condition [0x00007f81fc2bb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x000000074d9008e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "java-sdk-http-connection-reaper" daemon prio=10 tid=0x00007f820837e000 nid=0x5596 waiting on condition [0x00007f81fc3bc000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:112)
> "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon prio=10 tid=0x00007f8208352800 nid=0x5594 in Object.wait() [0x00007f824cc13000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x0000000756803100> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> 	- locked <0x0000000756803100> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> 	at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-client-0" daemon prio=10 tid=0x00007f8208110800 nid=0x5593 runnable [0x00007f824ca11000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x0000000756803238> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x0000000756803258> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000007568031f0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-client-0" daemon prio=10 tid=0x00007f820803b800 nid=0x5578 runnable [0x00007f824c704000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000007568033e0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x0000000756803400> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000756803398> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "driver-heartbeater" daemon prio=10 tid=0x00007f8200047800 nid=0x5573 waiting on condition [0x00007f81fdefb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000007568036b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-server-0" daemon prio=10 tid=0x00007f8200044000 nid=0x5572 runnable [0x00007f81fdffc000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000007568038c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000007568038e0> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000756803878> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-client-0" daemon prio=10 tid=0x000000000222c000 nid=0x5571 runnable [0x00007f824c1ff000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x0000000756803a68> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x0000000756803a88> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x0000000756803a20> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "netty-rpc-env-timeout" daemon prio=10 tid=0x00007f8285248000 nid=0x5570 waiting on condition [0x00007f824c300000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000756803b80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "dispatcher-event-loop-3" daemon prio=10 tid=0x00007f82851f4800 nid=0x556e waiting on condition [0x00007f824c502000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000756802418> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "dispatcher-event-loop-2" daemon prio=10 tid=0x00007f82851f3800 nid=0x556d waiting on condition [0x00007f824c805000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000756802418> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "dispatcher-event-loop-1" daemon prio=10 tid=0x00007f82851f3000 nid=0x556c waiting on condition [0x00007f824cf15000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000756802418> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "dispatcher-event-loop-0" daemon prio=10 tid=0x00007f82851f2000 nid=0x556b waiting on condition [0x00007f824c906000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000756802418> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "threadDeathWatcher-2-1" daemon prio=10 tid=0x00007f820400e000 nid=0x5567 waiting on condition [0x00007f824c603000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> 	at java.lang.Thread.run(Thread.java:745)
> "Service Thread" daemon prio=10 tid=0x00007f82842ae000 nid=0x555a runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f82842ab000 nid=0x5559 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f82842a9000 nid=0x5558 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=10 tid=0x00007f82842a6800 nid=0x5557 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f82842a4800 nid=0x5556 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=10 tid=0x00007f8284282800 nid=0x5555 in Object.wait() [0x00007f8280dfc000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x00000007568040a0> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> 	- locked <0x00000007568040a0> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" daemon prio=10 tid=0x00007f8284280800 nid=0x5554 in Object.wait() [0x00007f8280efd000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x00000007568040e0> (a java.lang.ref.Reference$Lock)
> 	at java.lang.Object.wait(Object.java:503)
> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> 	- locked <0x00000007568040e0> (a java.lang.ref.Reference$Lock)
> "main" prio=10 tid=0x00007f8284021000 nid=0x5547 waiting on condition [0x00007f828da05000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000756804ac8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468)
> 	at org.apache.spark.rpc.netty.Dispatcher.awaitTermination(Dispatcher.scala:180)
> 	at org.apache.spark.rpc.netty.NettyRpcEnv.awaitTermination(NettyRpcEnv.scala:273)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:217)
> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:71)
> 	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:70)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> 	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:174)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:270)
> 	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
> "VM Thread" prio=10 tid=0x00007f828427c000 nid=0x5553 runnable 
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f8284035800 nid=0x5548 runnable 
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f8284037800 nid=0x5549 runnable 
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f8284039000 nid=0x554a runnable 
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f828403b000 nid=0x554b runnable 
> "G1 Main Concurrent Mark GC Thread" prio=10 tid=0x00007f828404f800 nid=0x5551 runnable 
> "Gang worker#0 (G1 Parallel Marking Threads)" prio=10 tid=0x00007f8284062000 nid=0x5552 runnable 
> "G1 Concurrent Refinement Thread#0" prio=10 tid=0x00007f8284045800 nid=0x5550 runnable 
> "G1 Concurrent Refinement Thread#1" prio=10 tid=0x00007f8284043800 nid=0x554f runnable 
> "G1 Concurrent Refinement Thread#2" prio=10 tid=0x00007f8284041800 nid=0x554e runnable 
> "G1 Concurrent Refinement Thread#3" prio=10 tid=0x00007f828403f800 nid=0x554d runnable 
> "G1 Concurrent Refinement Thread#4" prio=10 tid=0x00007f828403e000 nid=0x554c runnable 
> "VM Periodic Task Thread" prio=10 tid=0x00007f82842b8800 nid=0x555b waiting on condition 
> JNI global references: 358
> {code}
> Jstack of driver:
> {code}
> Full thread dump OpenJDK 64-Bit Server VM (24.111-b01 mixed mode):
> "DestroyJavaVM" prio=10 tid=0x00007f0878012000 nid=0x342b waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "java-sdk-progress-listener-callback-thread" daemon prio=10 tid=0x00007f0844007000 nid=0x5da1 waiting on condition [0x00007f07dee9f000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c813a350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "task-result-getter-3" daemon prio=10 tid=0x00007f082c017800 nid=0x3715 waiting on condition [0x00007f07df4a5000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0275eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "task-result-getter-2" daemon prio=10 tid=0x00007f083400e000 nid=0x370c waiting on condition [0x00007f07deb9c000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0275eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "task-result-getter-1" daemon prio=10 tid=0x00007f0830014800 nid=0x3709 waiting on condition [0x00007f07df3a4000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0275eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "task-result-getter-0" daemon prio=10 tid=0x00007f083400d000 nid=0x36c2 waiting on condition [0x00007f07dea9b000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0275eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-server-3" daemon prio=10 tid=0x00007f0844005800 nid=0x34a6 runnable [0x00007f07e49af000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c027f6b0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000c027f710> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c027f608> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-server-2" daemon prio=10 tid=0x00007f0844004800 nid=0x34a3 runnable [0x00007f07e84c6000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c0283cb8> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000c0284d28> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c0283c20> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-server-1" daemon prio=10 tid=0x00007f0844002000 nid=0x34a0 runnable [0x00007f07d8f01000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c0285398> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000c02853f8> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c0285300> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-server-3" daemon prio=10 tid=0x00007f0820004800 nid=0x349e runnable [0x00007f07db004000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c02e3d48> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000c02e4db8> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c02e3cb0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-server-2" daemon prio=10 tid=0x00007f0820004000 nid=0x349c runnable [0x00007f07df5a6000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c02e5428> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000c02e5488> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c02e5390> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-server-1" daemon prio=10 tid=0x00007f0820003000 nid=0x349b runnable [0x00007f07df6a7000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c02f0af0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000c02f1b60> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c02f0a58> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "java-sdk-http-connection-reaper" daemon prio=10 tid=0x00007f0879fcb000 nid=0x348c waiting on condition [0x00007f07dd314000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:112)
> "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon prio=10 tid=0x00007f0878ad7800 nid=0x348b in Object.wait() [0x00007f07de697000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> 	- locked <0x00000000c6b4da98> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> 	at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063)
> 	at java.lang.Thread.run(Thread.java:745)
> "ForkJoinPool-1-worker-21" daemon prio=10 tid=0x00007f0808002000 nid=0x347e waiting on condition [0x00007f07df1a2000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c6d4db68> (a scala.concurrent.forkjoin.ForkJoinPool)
> 	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "threadDeathWatcher-2-1" daemon prio=10 tid=0x00007f0860024800 nid=0x3470 waiting on condition [0x00007f07e7eb7000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
> 	at java.lang.Thread.sleep(Native Method)
> 	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> 	at java.lang.Thread.run(Thread.java:745)
> "pool-19-thread-1" daemon prio=10 tid=0x00007f0879d9f800 nid=0x346a waiting on condition [0x00007f07e85c7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c2825fc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "SparkListenerBus" daemon prio=10 tid=0x00007f0879a4b800 nid=0x3468 waiting on condition [0x00007f07e86c8000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0119688> (a java.util.concurrent.Semaphore$NonfairSync)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> 	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:80)
> 	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> 	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
> 	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> 	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
> 	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1249)
> 	at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)
> "context-cleaner-periodic-gc" daemon prio=10 tid=0x00007f0879a45800 nid=0x3467 waiting on condition [0x00007f07e87c9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0544fe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "Spark Context Cleaner" daemon prio=10 tid=0x00007f0879a44800 nid=0x3466 in Object.wait() [0x00007f07e88ca000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> 	- locked <0x00000000c0544b88> (a java.lang.ref.ReferenceQueue$Lock)
> 	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:175)
> 	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1249)
> 	at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:172)
> 	at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:67)
> "Scheduler-214815552" prio=10 tid=0x00007f0804018000 nid=0x3465 waiting on condition [0x00007f07e89cb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c031b528> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-server-0" daemon prio=10 tid=0x00007f087996d800 nid=0x3464 runnable [0x00007f07e8acc000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c0290de0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000c029f290> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c0290d48> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "MesosCoarseGrainedSchedulerBackend-mesos-driver" daemon prio=10 tid=0x00007f08798a4000 nid=0x3463 runnable [0x00007f07e8dd3000]
>    java.lang.Thread.State: RUNNABLE
> 	at org.apache.mesos.MesosSchedulerDriver.join(Native Method)
> 	at org.apache.mesos.MesosSchedulerDriver.run(MesosSchedulerDriver.java:168)
> 	at org.apache.spark.scheduler.cluster.mesos.MesosSchedulerUtils$$anon$1.run(MesosSchedulerUtils.scala:120)
> "driver-revive-thread" daemon prio=10 tid=0x00007f0830003000 nid=0x3459 waiting on condition [0x00007f07ed6dd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c02775c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "dag-scheduler-event-loop" daemon prio=10 tid=0x00007f0879873800 nid=0x3458 waiting on condition [0x00007f07ed7de000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c026e200> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
> 	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)
> "Timer-0" daemon prio=10 tid=0x00007f0879821000 nid=0x3457 in Object.wait() [0x00007f083c846000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	- waiting on <0x00000000c0267430> (a java.util.TaskQueue)
> 	at java.lang.Object.wait(Object.java:503)
> 	at java.util.TimerThread.mainLoop(Timer.java:526)
> 	- locked <0x00000000c0267430> (a java.util.TaskQueue)
> 	at java.util.TimerThread.run(Timer.java:505)
> "netty-rpc-env-timeout" daemon prio=10 tid=0x00007f080400e800 nid=0x3456 waiting on condition [0x00007f083c645000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c02e1338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "heartbeat-receiver-event-loop-thread" daemon prio=10 tid=0x00007f0834005800 nid=0x3455 waiting on condition [0x00007f083e277000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c02764f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "SparkUI-42" daemon prio=10 tid=0x00007f08797ba000 nid=0x3453 waiting on condition [0x00007f083da2f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0312500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> 	at java.lang.Thread.run(Thread.java:745)
> "SparkUI-41" daemon prio=10 tid=0x00007f08797b8000 nid=0x3452 waiting on condition [0x00007f083db30000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0312500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> 	at java.lang.Thread.run(Thread.java:745)
> "SparkUI-40" daemon prio=10 tid=0x00007f08797b6000 nid=0x3451 waiting on condition [0x00007f083dc31000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0312500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> 	at java.lang.Thread.run(Thread.java:745)
> "SparkUI-39" daemon prio=10 tid=0x00007f08797b3800 nid=0x3450 waiting on condition [0x00007f083dd32000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0312500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> 	at java.lang.Thread.run(Thread.java:745)
> "SparkUI-38-acceptor-0@6100756c-ServerConnector@651d3734{HTTP/1.1}{0.0.0.0:4040}" daemon prio=10 tid=0x00007f08797b1800 nid=0x344f runnable [0x00007f083de33000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> 	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> 	- locked <0x00000000c0312cc8> (a java.lang.Object)
> 	at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377)
> 	at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> 	at java.lang.Thread.run(Thread.java:745)
> "SparkUI-37" daemon prio=10 tid=0x00007f08797af800 nid=0x344e waiting on condition [0x00007f083df34000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0312500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> 	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> 	at java.lang.Thread.run(Thread.java:745)
> "SparkUI-36-selector-ServerConnectorManager@13aba9a3/1" daemon prio=10 tid=0x00007f08797ae000 nid=0x344d runnable [0x00007f083e035000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c031b668> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000c031b658> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c03130a0> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> 	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
> 	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
> 	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> 	at java.lang.Thread.run(Thread.java:745)
> "SparkUI-35-selector-ServerConnectorManager@13aba9a3/0" daemon prio=10 tid=0x00007f08797ac800 nid=0x344c runnable [0x00007f083e136000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c0315cd0> (a sun.nio.ch.Util$2)
> 	- locked <0x00000000c0315cc0> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c0315b38> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> 	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601)
> 	at org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550)
> 	at org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> 	at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> 	at java.lang.Thread.run(Thread.java:745)
> "map-output-dispatcher-7" daemon prio=10 tid=0x00007f0879543000 nid=0x344a waiting on condition [0x00007f083e378000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0268050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "map-output-dispatcher-6" daemon prio=10 tid=0x00007f0879540800 nid=0x3449 waiting on condition [0x00007f083e479000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0268050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "map-output-dispatcher-5" daemon prio=10 tid=0x00007f087953e800 nid=0x3448 waiting on condition [0x00007f083e57a000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0268050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "map-output-dispatcher-4" daemon prio=10 tid=0x00007f087953c800 nid=0x3447 waiting on condition [0x00007f083e67b000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0268050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "map-output-dispatcher-3" daemon prio=10 tid=0x00007f087953b000 nid=0x3446 waiting on condition [0x00007f083e77c000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0268050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "map-output-dispatcher-2" daemon prio=10 tid=0x00007f0879539800 nid=0x3445 waiting on condition [0x00007f083e87d000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0268050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "map-output-dispatcher-1" daemon prio=10 tid=0x00007f0879532800 nid=0x3444 waiting on condition [0x00007f083e97e000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0268050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "map-output-dispatcher-0" daemon prio=10 tid=0x00007f0879531000 nid=0x3443 waiting on condition [0x00007f083ea7f000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c0268050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "shuffle-server-0" daemon prio=10 tid=0x00007f0879515800 nid=0x3442 runnable [0x00007f083eb80000]
>    java.lang.Thread.State: RUNNABLE
> 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> 	- locked <0x00000000c02f24c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
> 	- locked <0x00000000c02fdbf0> (a java.util.Collections$UnmodifiableSet)
> 	- locked <0x00000000c02f2428> (a sun.nio.ch.EPollSelectorImpl)
> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> 	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> "dispatcher-event-loop-3" daemon prio=10 tid=0x00007f087937c800 nid=0x3440 waiting on condition [0x00007f083ec8b000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c02c10b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "dispatcher-event-loop-2" daemon prio=10 tid=0x00007f087937a800 nid=0x343f waiting on condition [0x00007f083ed8c000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c02c10b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "dispatcher-event-loop-1" daemon prio=10 tid=0x00007f0879379800 nid=0x343e waiting on condition [0x00007f083ee8d000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c02c10b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "dispatcher-event-loop-0" daemon prio=10 tid=0x00007f0879378800 nid=0x343d waiting on condition [0x00007f083f190000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x00000000c02c10b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> "Service Thread" daemon prio=10 tid=0x00007f08780b6000 nid=0x3436 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f08780b4000 nid=0x3435 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f08780b1000 nid=0x3434 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=10 tid=0x00007f08780ae800 nid=0x3433 waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=10 tid=0x00007f0878084000 nid=0x3432 in Object.wait() [0x00007f0874af9000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> 	- locked <0x00000000c0015cc8> (a java.lang.ref.ReferenceQueue$Lock)
> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" daemon prio=10 tid=0x00007f0878082000 nid=0x3431 in Object.wait() [0x00007f0874bfa000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Object.wait(Object.java:503)
> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> 	- locked <0x00000000c0015c98> (a java.lang.ref.Reference$Lock)
> "VM Thread" prio=10 tid=0x00007f087807d800 nid=0x3430 runnable 
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f0878028000 nid=0x342c runnable 
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f087802a000 nid=0x342d runnable 
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f087802c000 nid=0x342e runnable 
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f087802d800 nid=0x342f runnable 
> "VM Periodic Task Thread" prio=10 tid=0x00007f08780c1000 nid=0x3437 waiting on condition 
> JNI global references: 309
> Heap
>  PSYoungGen      total 136192K, used 39381K [0x00000000eaa80000, 0x00000000f6b00000, 0x0000000100000000)
>   eden space 135680K, 29% used [0x00000000eaa80000,0x00000000ed0f56b0,0x00000000f2f00000)
>   from space 512K, 0% used [0x00000000f2f00000,0x00000000f2f00000,0x00000000f2f80000)
>   to   space 30720K, 0% used [0x00000000f4d00000,0x00000000f4d00000,0x00000000f6b00000)
>  ParOldGen       total 243200K, used 133233K [0x00000000c0000000, 0x00000000ced80000, 0x00000000eaa80000)
>   object space 243200K, 54% used [0x00000000c0000000,0x00000000c821c438,0x00000000ced80000)
>  PSPermGen       total 119296K, used 74568K [0x00000000b0000000, 0x00000000b7480000, 0x00000000c0000000)
>   object space 119296K, 62% used [0x00000000b0000000,0x00000000b48d2208,0x00000000b7480000)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message