Return-Path: X-Original-To: apmail-spark-commits-archive@minotaur.apache.org Delivered-To: apmail-spark-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 16E57187BE for ; Fri, 18 Mar 2016 19:40:05 +0000 (UTC) Received: (qmail 96413 invoked by uid 500); 18 Mar 2016 19:40:05 -0000 Delivered-To: apmail-spark-commits-archive@spark.apache.org Received: (qmail 96383 invoked by uid 500); 18 Mar 2016 19:40:05 -0000 Mailing-List: contact commits-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list commits@spark.apache.org Received: (qmail 96374 invoked by uid 99); 18 Mar 2016 19:40:05 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2016 19:40:05 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id D96FDDFA40; Fri, 18 Mar 2016 19:40:04 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: vanzin@apache.org To: commits@spark.apache.org Message-Id: <8f305b27e1444717bcb146d8715ed0b2@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: spark git commit: [SPARK-13885][YARN] Fix attempt id regression for Spark running on Yarn Date: Fri, 18 Mar 2016 19:40:04 +0000 (UTC) Repository: spark Updated Branches: refs/heads/master 9c23c818c -> 353778216 [SPARK-13885][YARN] Fix attempt id regression for Spark running on Yarn ## What changes were proposed in this pull request? This regression is introduced in #9182, previously attempt id is simply as counter "1" or "2". With the change of #9182, it is changed to full name as "appattemtp-xxx-00001", this will affect all the parts which uses this attempt id, like event log file name, history server app url link. So here change it back to the counter to keep consistent with previous code. Also revert back this patch #11518, this patch fix the url link of history log according to the new way of attempt id, since here we change back to the previous way, so this patch is not necessary, here to revert it. Also clean "spark.yarn.app.id" and "spark.yarn.app.attemptId", since it is useless now. ## How was this patch tested? Test it with unit test and manually test different scenario: 1. application running in yarn-client mode. 2. application running in yarn-cluster mode. 3. application running in yarn-cluster mode with multiple attempts. Checked both the event log file name and url link. CC vanzin tgravescs , please help to review, thanks a lot. Author: jerryshao Closes #11721 from jerryshao/SPARK-13885. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/35377821 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/35377821 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/35377821 Branch: refs/heads/master Commit: 3537782168aa9278ac4add1a25afac0ec6e17085 Parents: 9c23c81 Author: jerryshao Authored: Fri Mar 18 12:39:49 2016 -0700 Committer: Marcelo Vanzin Committed: Fri Mar 18 12:39:49 2016 -0700 ---------------------------------------------------------------------- .../spark/ui/static/historypage-template.html | 4 ++-- .../org/apache/spark/ui/static/historypage.js | 19 ++----------------- .../spark/deploy/yarn/ApplicationMaster.scala | 7 ++----- .../scheduler/cluster/YarnSchedulerBackend.scala | 3 ++- 4 files changed, 8 insertions(+), 25 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/35377821/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html index 5a7a252..a2b3826 100644 --- a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html +++ b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html @@ -64,10 +64,10 @@ {{#applications}} - {{id}} + {{id}} {{name}} {{#attempts}} - {{attemptId}} + {{attemptId}} {{startTime}} {{endTime}} {{duration}} http://git-wip-us.apache.org/repos/asf/spark/blob/35377821/core/src/main/resources/org/apache/spark/ui/static/historypage.js ---------------------------------------------------------------------- diff --git a/core/src/main/resources/org/apache/spark/ui/static/historypage.js b/core/src/main/resources/org/apache/spark/ui/static/historypage.js index 6096513..ef89a9a 100644 --- a/core/src/main/resources/org/apache/spark/ui/static/historypage.js +++ b/core/src/main/resources/org/apache/spark/ui/static/historypage.js @@ -123,28 +123,13 @@ $(document).ready(function() { if (app["attempts"].length > 1) { hasMultipleAttempts = true; } - - var maxAttemptId = null + var num = app["attempts"].length; for (j in app["attempts"]) { var attempt = app["attempts"][j]; - if (attempt['attemptId'] != null) { - if (maxAttemptId == null || attempt['attemptId'] > maxAttemptId) { - maxAttemptId = attempt['attemptId'] - } - } - attempt["startTime"] = formatDate(attempt["startTime"]); attempt["endTime"] = formatDate(attempt["endTime"]); attempt["lastUpdated"] = formatDate(attempt["lastUpdated"]); - - var url = null - if (maxAttemptId == null) { - url = "history/" + id + "/" - } else { - url = "history/" + id + "/" + maxAttemptId + "/" - } - - var app_clone = {"id" : id, "name" : name, "url" : url, "attempts" : [attempt]}; + var app_clone = {"id" : id, "name" : name, "num" : num, "attempts" : [attempt]}; array.push(app_clone); } } http://git-wip-us.apache.org/repos/asf/spark/blob/35377821/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---------------------------------------------------------------------- diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala index 84445d6..e941089 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala @@ -137,12 +137,9 @@ private[spark] class ApplicationMaster( System.setProperty("spark.master", "yarn") System.setProperty("spark.submit.deployMode", "cluster") - // Propagate the application ID so that YarnClusterSchedulerBackend can pick it up. + // Set this internal configuration if it is running on cluster mode, this + // configuration will be checked in SparkContext to avoid misuse of yarn cluster mode. System.setProperty("spark.yarn.app.id", appAttemptId.getApplicationId().toString()) - - // Propagate the attempt if, so that in case of event logging, - // different attempt's logs gets created in different directory - System.setProperty("spark.yarn.app.attemptId", appAttemptId.getAttemptId().toString()) } logInfo("ApplicationAttemptId: " + appAttemptId) http://git-wip-us.apache.org/repos/asf/spark/blob/35377821/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala ---------------------------------------------------------------------- diff --git a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala index 0cc158b..a878163 100644 --- a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala +++ b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala @@ -96,11 +96,12 @@ private[spark] abstract class YarnSchedulerBackend( /** * Get the attempt ID for this run, if the cluster manager supports multiple * attempts. Applications run in client mode will not have attempt IDs. + * This attempt ID only includes attempt counter, like "1", "2". * * @return The application attempt id, if available. */ override def applicationAttemptId(): Option[String] = { - attemptId.map(_.toString) + attemptId.map(_.getAttemptId.toString) } /** --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org For additional commands, e-mail: commits-help@spark.apache.org