spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: [SPARK-6314] [CORE] handle JsonParseException for history server
Date Tue, 28 Apr 2015 19:37:29 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.3 80ba82e8e -> 37fb5a968


[SPARK-6314] [CORE] handle JsonParseException for history server

This is handled in the same way with [SPARK-6197](https://issues.apache.org/jira/browse/SPARK-6197).
The result of this PR is that exception showed in history server log will be replaced by a
warning, and the application that with un-complete history log file will be listed on history
server webUI

Author: Zhang, Liye <liye.zhang@intel.com>

Closes #5736 from liyezhang556520/SPARK-6314 and squashes the following commits:

b8d2d88 [Zhang, Liye] handle JsonParseException for history server


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/37fb5a96
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/37fb5a96
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/37fb5a96

Branch: refs/heads/branch-1.3
Commit: 37fb5a96812d9a8b2fb4b9492feb4c91f4f64f91
Parents: 80ba82e
Author: Zhang, Liye <liye.zhang@intel.com>
Authored: Tue Apr 28 12:33:48 2015 -0700
Committer: Andrew Or <andrew@databricks.com>
Committed: Tue Apr 28 12:37:25 2015 -0700

----------------------------------------------------------------------
 .../org/apache/spark/deploy/history/FsHistoryProvider.scala     | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/37fb5a96/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 554ce13..c55d258 100644
--- a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -256,8 +256,9 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis
       }
     try {
       val appListener = new ApplicationEventListener
+      val appCompleted = isApplicationCompleted(eventLog)
       bus.addListener(appListener)
-      bus.replay(logInput, logPath.toString)
+      bus.replay(logInput, logPath.toString, !appCompleted)
       new FsApplicationHistoryInfo(
         logPath.getName(),
         appListener.appId.getOrElse(logPath.getName()),
@@ -266,7 +267,7 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis
         appListener.endTime.getOrElse(-1L),
         getModificationTime(eventLog).get,
         appListener.sparkUser.getOrElse(NOT_STARTED),
-        isApplicationCompleted(eventLog))
+        appCompleted)
     } finally {
       logInput.close()
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message