spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject spark git commit: Add a note for context termination for History server on Yarn
Date Thu, 26 Feb 2015 22:21:04 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.0 f74bccbe3 -> 14e042b65

Add a note for context termination for History server on Yarn

The history server on Yarn only shows completed jobs. This adds a note concerning the needed
explicit context termination at the end of a spark job which is a best practice anyway.
Related to SPARK-2972 and SPARK-3458

Author: moussa taifi <>

Closes #4721 from moutai/add-history-server-note-for-closing-the-spark-context and squashes
the following commits:

9f5b6c3 [moussa taifi] Fix upper case typo for YARN
3ad3db4 [moussa taifi] Add context termination for History server on Yarn

(cherry picked from commit c871e2dae0182e914135560d14304242e1f97f7e)
Signed-off-by: Andrew Or <>


Branch: refs/heads/branch-1.0
Commit: 14e042b65f8b927e432519c668e15f65a062dde7
Parents: f74bccb
Author: moussa taifi <>
Authored: Thu Feb 26 14:19:43 2015 -0800
Committer: Andrew Or <>
Committed: Thu Feb 26 14:21:02 2015 -0800

 docs/ | 2 ++
 1 file changed, 2 insertions(+)
diff --git a/docs/ b/docs/
index 8578a39..60d9c3f 100644
--- a/docs/
+++ b/docs/
@@ -133,6 +133,8 @@ represents an application's event logs. This creates a web interface at
 Note that in all of these UIs, the tables are sortable by clicking their headers,
 making it easy to identify slow tasks, data skew, etc.
+Note that the history server only displays completed Spark jobs. One way to signal the completion
of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the
`with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show
the job history on the UI.
 # Metrics
 Spark has a configurable metrics system based on the 

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message