spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject spark git commit: [SPARK-21176][WEB UI] Limit number of selector threads for admin ui proxy servlets to 8
Date Fri, 30 Jun 2017 03:16:29 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 8de67e369 -> c6ba64793

[SPARK-21176][WEB UI] Limit number of selector threads for admin ui proxy servlets to 8

## What changes were proposed in this pull request?
Please see also

This change limits the number of selector threads that jetty creates to maximum 8 per proxy
servlet (Jetty default is number of processors / 2).
The newHttpClient for Jettys ProxyServlet class is overwritten to avoid the Jetty defaults
(which are designed for high-performance http servers).
Once is available, the code could be
cleaned up to avoid the method override.

I really need this on v2.1.1 - what is the best way for a backport automatic merge works fine)?
Shall I create another PR?

## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
The patch was tested manually on a Spark cluster with a head node that has 88 processors using
JMX to verify that the number of selector threads is now limited to 8 per proxy.

gurvindersingh zsxwing can you please review the change?

Author: IngoSchuster <>
Author: Ingo Schuster <>

Closes #18437 from IngoSchuster/master.

(cherry picked from commit 88a536babf119b7e331d02aac5d52b57658803bf)
Signed-off-by: Wenchen Fan <>


Branch: refs/heads/branch-2.2
Commit: c6ba647935e9108d139cc8914091790917567ad7
Parents: 8de67e3
Author: IngoSchuster <>
Authored: Fri Jun 30 11:16:09 2017 +0800
Committer: Wenchen Fan <>
Committed: Fri Jun 30 11:16:19 2017 +0800

 .../src/main/scala/org/apache/spark/ui/JettyUtils.scala | 12 ++++++++++++
 1 file changed, 12 insertions(+)
diff --git a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
index edf328b..b9371c7 100644
--- a/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
+++ b/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala
@@ -26,6 +26,8 @@ import scala.language.implicitConversions
 import scala.xml.Node
 import org.eclipse.jetty.client.api.Response
+import org.eclipse.jetty.client.HttpClient
+import org.eclipse.jetty.client.http.HttpClientTransportOverHTTP
 import org.eclipse.jetty.proxy.ProxyServlet
 import org.eclipse.jetty.server._
 import org.eclipse.jetty.server.handler._
@@ -208,6 +210,16 @@ private[spark] object JettyUtils extends Logging {
+      override def newHttpClient(): HttpClient = {
+        // SPARK-21176: Use the Jetty logic to calculate the number of selector threads (#CPUs/2),
+        // but limit it to 8 max.
+        // Otherwise, it might happen that we exhaust the threadpool since in reverse proxy
+        // a proxy is instantiated for each executor. If the head node has many processors,
+        // can quickly add up to an unreasonably high number of threads.
+        val numSelectors = math.max(1, math.min(8, Runtime.getRuntime().availableProcessors()
/ 2))
+        new HttpClient(new HttpClientTransportOverHTTP(numSelectors), null)
+      }
       override def filterServerResponseHeader(
           clientRequest: HttpServletRequest,
           serverResponse: Response,

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message