spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-6868][YARN] Fix broken container log link on executor page when HTTPS_ONLY.
Date Mon, 13 Apr 2015 11:12:28 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.3 8d4176132 -> 8e5caa227


[SPARK-6868][YARN] Fix broken container log link on executor page when HTTPS_ONLY.

Correct http schema in YARN container log link in Spark UI when container logs when YARN is
configured to be HTTPS_ONLY.

Uses the same logic as the YARN jobtracker webapp. Entry point is [JobBlock](https://github.com/apache/hadoop/blob/e1109fb65608a668cd53dc324dadc6f63a74eeb9/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java#L108)
and logic is in [MRWebAppUtil](https://github.com/apache/hadoop/blob/e1109fb65608a668cd53dc324dadc6f63a74eeb9/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRWebAppUtil.java#L75).

I chose to migrate the logic over instead of importing MRWebAppUtil(but can update the PR
to do so) since the class is designated as private and the logic was straightforward.

Author: Dean Chen <deanchen5@gmail.com>

Closes #5477 from deanchen/container-url and squashes the following commits:

91d3090 [Dean Chen] Correct http schema in YARN container log link in Spark UI when container
logs when YARN is configured to be HTTPS_ONLY.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8e5caa22
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8e5caa22
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8e5caa22

Branch: refs/heads/branch-1.3
Commit: 8e5caa227238f33a7b824ffe953a97245b1ad864
Parents: 8d41761
Author: Dean Chen <deanchen5@gmail.com>
Authored: Mon Apr 13 12:08:55 2015 +0100
Committer: Sean Owen <sowen@cloudera.com>
Committed: Mon Apr 13 12:11:28 2015 +0100

----------------------------------------------------------------------
 .../apache/spark/deploy/yarn/ExecutorRunnable.scala    | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/8e5caa22/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
----------------------------------------------------------------------
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
index c1d3f73..8cf8512 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
@@ -290,10 +290,19 @@ class ExecutorRunnable(
       YarnSparkHadoopUtil.setEnvFromInputString(env, userEnvs)
     }
 
+    // lookup appropriate http scheme for container log urls
+    val yarnHttpPolicy = yarnConf.get(
+      YarnConfiguration.YARN_HTTP_POLICY_KEY,
+      YarnConfiguration.YARN_HTTP_POLICY_DEFAULT
+    )
+    val httpScheme = if (yarnHttpPolicy == "HTTPS_ONLY") "https://" else "http://"
+
     // Add log urls
     sys.env.get("SPARK_USER").foreach { user =>
-      val baseUrl = "http://%s/node/containerlogs/%s/%s"
-        .format(container.getNodeHttpAddress, ConverterUtils.toString(container.getId), user)
+      val containerId = ConverterUtils.toString(container.getId)
+      val address = container.getNodeHttpAddress
+      val baseUrl = s"$httpScheme$address/node/containerlogs/$containerId/$user"
+
       env("SPARK_LOG_URL_STDERR") = s"$baseUrl/stderr?start=0"
       env("SPARK_LOG_URL_STDOUT") = s"$baseUrl/stdout?start=0"
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message