spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zsxw...@apache.org
Subject spark git commit: [SPARK-12608][STREAMING] Remove submitJobThreadPool since submitJob doesn't create a separate thread to wait for the job result
Date Mon, 04 Jan 2016 19:00:19 GMT
Repository: spark
Updated Branches:
  refs/heads/master b504b6a90 -> 43706bf8b


[SPARK-12608][STREAMING] Remove submitJobThreadPool since submitJob doesn't create a separate
thread to wait for the job result

Before #9264, submitJob would create a separate thread to wait for the job result. `submitJobThreadPool`
was a workaround in `ReceiverTracker` to run these waiting-job-result threads. Now #9264 has
been merged to master and resolved this blocking issue, `submitJobThreadPool` can be removed
now.

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #10560 from zsxwing/remove-submitJobThreadPool.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/43706bf8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/43706bf8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/43706bf8

Branch: refs/heads/master
Commit: 43706bf8bdfe08010bb11848788e0718d15363b3
Parents: b504b6a
Author: Shixiong Zhu <shixiong@databricks.com>
Authored: Mon Jan 4 11:00:15 2016 -0800
Committer: Shixiong Zhu <shixiong@databricks.com>
Committed: Mon Jan 4 11:00:15 2016 -0800

----------------------------------------------------------------------
 .../apache/spark/streaming/scheduler/ReceiverTracker.scala    | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/43706bf8/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
----------------------------------------------------------------------
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
index 9ddf176..678f1dc 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
@@ -435,10 +435,6 @@ class ReceiverTracker(ssc: StreamingContext, skipReceiverLaunch: Boolean
= false
   /** RpcEndpoint to receive messages from the receivers. */
   private class ReceiverTrackerEndpoint(override val rpcEnv: RpcEnv) extends ThreadSafeRpcEndpoint
{
 
-    // TODO Remove this thread pool after https://github.com/apache/spark/issues/7385 is
merged
-    private val submitJobThreadPool = ExecutionContext.fromExecutorService(
-      ThreadUtils.newDaemonCachedThreadPool("submit-job-thread-pool"))
-
     private val walBatchingThreadPool = ExecutionContext.fromExecutorService(
       ThreadUtils.newDaemonCachedThreadPool("wal-batching-thread-pool"))
 
@@ -610,12 +606,11 @@ class ReceiverTracker(ssc: StreamingContext, skipReceiverLaunch: Boolean
= false
             logInfo(s"Restarting Receiver $receiverId")
             self.send(RestartReceiver(receiver))
           }
-      }(submitJobThreadPool)
+      }(ThreadUtils.sameThread)
       logInfo(s"Receiver ${receiver.streamId} started")
     }
 
     override def onStop(): Unit = {
-      submitJobThreadPool.shutdownNow()
       active = false
       walBatchingThreadPool.shutdown()
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message