spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: [SPARK-5344][WebUI] HistoryServer cannot recognize that inprogress file was renamed to completed file
Date Sun, 25 Jan 2015 23:36:45 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.2 f34c113f8 -> 8f55beeb5


[SPARK-5344][WebUI] HistoryServer cannot recognize that inprogress file was renamed to completed
file

`FsHistoryProvider` tries to update application status but if `checkForLogs` is called before
`.inprogress` file is renamed to completed file, the file is not recognized as completed.

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #4132 from sarutak/SPARK-5344 and squashes the following commits:

9658008 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5344
d2c72b6 [Kousuke Saruta] Fixed update issue of FsHistoryProvider

(cherry picked from commit 8f5c827b01026bf45fc774ed7387f11a941abea8)
Signed-off-by: Andrew Or <andrew@databricks.com>

Conflicts:
	core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8f55beeb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8f55beeb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8f55beeb

Branch: refs/heads/branch-1.2
Commit: 8f55beeb51e6ea72e63af3f276497f61dd24d09b
Parents: f34c113
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Authored: Sun Jan 25 15:34:20 2015 -0800
Committer: Andrew Or <andrew@databricks.com>
Committed: Sun Jan 25 15:36:24 2015 -0800

----------------------------------------------------------------------
 .../org/apache/spark/deploy/history/FsHistoryProvider.scala      | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/8f55beeb/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 82a54db..4671ca2 100644
--- a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -197,7 +197,9 @@ private[history] class FsHistoryProvider(conf: SparkConf) extends ApplicationHis
       if (!logInfos.isEmpty) {
         val newApps = new mutable.LinkedHashMap[String, FsApplicationHistoryInfo]()
         def addIfAbsent(info: FsApplicationHistoryInfo) = {
-          if (!newApps.contains(info.id)) {
+          if (!newApps.contains(info.id) ||
+              newApps(info.id).logPath.endsWith(EventLoggingListener.IN_PROGRESS) &&
+              !info.logPath.endsWith(EventLoggingListener.IN_PROGRESS)) {
             newApps += (info.id -> info)
           }
         }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message