Return-Path: X-Original-To: apmail-spark-commits-archive@minotaur.apache.org Delivered-To: apmail-spark-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 86A9010F8B for ; Tue, 1 Apr 2014 06:01:25 +0000 (UTC) Received: (qmail 47881 invoked by uid 500); 1 Apr 2014 06:01:25 -0000 Delivered-To: apmail-spark-commits-archive@spark.apache.org Received: (qmail 47854 invoked by uid 500); 1 Apr 2014 06:01:23 -0000 Mailing-List: contact commits-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@spark.apache.org Delivered-To: mailing list commits@spark.apache.org Received: (qmail 47829 invoked by uid 99); 1 Apr 2014 06:01:22 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Apr 2014 06:01:22 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id C43B391ACB5; Tue, 1 Apr 2014 06:01:21 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: pwendell@apache.org To: commits@spark.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: git commit: [Hot Fix #42] Persisted RDD disappears on storage page if re-used Date: Tue, 1 Apr 2014 06:01:21 +0000 (UTC) Repository: spark Updated Branches: refs/heads/master 94fe7fd4f -> ada310a9d [Hot Fix #42] Persisted RDD disappears on storage page if re-used If a previously persisted RDD is re-used, its information disappears from the Storage page. This is because the tasks associated with re-using the RDD do not report the RDD's blocks as updated (which is correct). On stage submit, however, we overwrite any existing information regarding that RDD with a fresh one, whether or not the information for the RDD already exists. Author: Andrew Or Closes #281 from andrewor14/ui-storage-fix and squashes the following commits: 408585a [Andrew Or] Fix storage UI bug Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ada310a9 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ada310a9 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ada310a9 Branch: refs/heads/master Commit: ada310a9d3d5419e101b24d9b41398f609da1ad3 Parents: 94fe7fd Author: Andrew Or Authored: Mon Mar 31 23:01:14 2014 -0700 Committer: Patrick Wendell Committed: Mon Mar 31 23:01:14 2014 -0700 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/ada310a9/core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala b/core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala index 4d8b01d..a7b24ff 100644 --- a/core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala +++ b/core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala @@ -84,7 +84,7 @@ private[ui] class BlockManagerListener(storageStatusListener: StorageStatusListe override def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted) = synchronized { val rddInfo = stageSubmitted.stageInfo.rddInfo - _rddInfoMap(rddInfo.id) = rddInfo + _rddInfoMap.getOrElseUpdate(rddInfo.id, rddInfo) } override def onStageCompleted(stageCompleted: SparkListenerStageCompleted) = synchronized {