spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: Add a note for context termination for History server on Yarn
Date Thu, 26 Feb 2015 22:20:37 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.3 fe7967483 -> 297c3ef82


Add a note for context termination for History server on Yarn

The history server on Yarn only shows completed jobs. This adds a note concerning the needed
explicit context termination at the end of a spark job which is a best practice anyway.
Related to SPARK-2972 and SPARK-3458

Author: moussa taifi <moutai10@gmail.com>

Closes #4721 from moutai/add-history-server-note-for-closing-the-spark-context and squashes
the following commits:

9f5b6c3 [moussa taifi] Fix upper case typo for YARN
3ad3db4 [moussa taifi] Add context termination for History server on Yarn

(cherry picked from commit c871e2dae0182e914135560d14304242e1f97f7e)
Signed-off-by: Andrew Or <andrew@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/297c3ef8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/297c3ef8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/297c3ef8

Branch: refs/heads/branch-1.3
Commit: 297c3ef826933b4a71d5a54950cabdb1d7d54613
Parents: fe79674
Author: moussa taifi <moutai10@gmail.com>
Authored: Thu Feb 26 14:19:43 2015 -0800
Committer: Andrew Or <andrew@databricks.com>
Committed: Thu Feb 26 14:20:36 2015 -0800

----------------------------------------------------------------------
 docs/monitoring.md | 2 ++
 1 file changed, 2 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/297c3ef8/docs/monitoring.md
----------------------------------------------------------------------
diff --git a/docs/monitoring.md b/docs/monitoring.md
index 7a5cadc..657bf67b 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -150,6 +150,8 @@ follows:
 Note that in all of these UIs, the tables are sortable by clicking their headers,
 making it easy to identify slow tasks, data skew, etc.
 
+Note that the history server only displays completed Spark jobs. One way to signal the completion
of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the
`with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show
the job history on the UI.
+
 # Metrics
 
 Spark has a configurable metrics system based on the 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message