spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: [SPARK-5703] AllJobsPage throws empty.max exception
Date Tue, 10 Feb 2015 05:19:01 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.2 515f65804 -> 53de2378e


[SPARK-5703] AllJobsPage throws empty.max exception

If you have a `SparkListenerJobEnd` event without the corresponding `SparkListenerJobStart`
event, then `JobProgressListener` will create an empty `JobUIData` with an empty `stageIds`
list. However, later in `AllJobsPage` we call `stageIds.max`. If this is empty, it will throw
an exception.

This crashed my history server.

Author: Andrew Or <andrew@databricks.com>

Closes #4490 from andrewor14/jobs-page-max and squashes the following commits:

21797d3 [Andrew Or] Check nonEmpty before calling max

(cherry picked from commit a95ed52157473fb0e42e910ee15270e7f0edf943)
Signed-off-by: Andrew Or <andrew@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/53de2378
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/53de2378
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/53de2378

Branch: refs/heads/branch-1.2
Commit: 53de2378e6ba7d72a5b5a91eee03c63f8fab62ff
Parents: 515f658
Author: Andrew Or <andrew@databricks.com>
Authored: Mon Feb 9 21:18:48 2015 -0800
Committer: Andrew Or <andrew@databricks.com>
Committed: Mon Feb 9 21:18:59 2015 -0800

----------------------------------------------------------------------
 core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/53de2378/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
index ea2d187..8ab26a7 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
@@ -43,7 +43,9 @@ private[ui] class AllJobsPage(parent: JobsTab) extends WebUIPage("") {
     }
 
     def makeRow(job: JobUIData): Seq[Node] = {
-      val lastStageInfo = listener.stageIdToInfo.get(job.stageIds.max)
+      val lastStageInfo = Option(job.stageIds)
+        .filter(_.nonEmpty)
+        .flatMap { ids => listener.stageIdToInfo.get(ids.max) }
       val lastStageData = lastStageInfo.flatMap { s =>
         listener.stageIdToData.get((s.stageId, s.attemptId))
       }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message