spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-17525][PYTHON] Remove SparkContext.clearFiles() from the PySpark API as it was removed from the Scala API prior to Spark 2.0.0
Date Wed, 14 Sep 2016 08:38:33 GMT
Repository: spark
Updated Branches:
  refs/heads/master def7c265f -> b5bfcddbf


[SPARK-17525][PYTHON] Remove SparkContext.clearFiles() from the PySpark API as it was removed
from the Scala API prior to Spark 2.0.0

## What changes were proposed in this pull request?

This pull request removes the SparkContext.clearFiles() method from the PySpark API as the
method was removed from the Scala API in 8ce645d4eeda203cf5e100c4bdba2d71edd44e6a. Using that
method in PySpark leads to an exception as PySpark tries to call the non-existent method on
the JVM side.

## How was this patch tested?

Existing tests (though none of them tested this particular method).

Author: Sami Jaktholm <sjakthol@outlook.com>

Closes #15081 from sjakthol/pyspark-sc-clearfiles.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b5bfcddb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b5bfcddb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b5bfcddb

Branch: refs/heads/master
Commit: b5bfcddbfbc2e79d3d0fbd43942716946e6c4ba3
Parents: def7c26
Author: Sami Jaktholm <sjakthol@outlook.com>
Authored: Wed Sep 14 09:38:30 2016 +0100
Committer: Sean Owen <sowen@cloudera.com>
Committed: Wed Sep 14 09:38:30 2016 +0100

----------------------------------------------------------------------
 python/pyspark/context.py | 8 --------
 1 file changed, 8 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/b5bfcddb/python/pyspark/context.py
----------------------------------------------------------------------
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index 6e9f24e..2744bb9 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -787,14 +787,6 @@ class SparkContext(object):
         """
         self._jsc.sc().addFile(path)
 
-    def clearFiles(self):
-        """
-        Clear the job's list of files added by L{addFile} or L{addPyFile} so
-        that they do not get downloaded to any new nodes.
-        """
-        # TODO: remove added .py or .zip files from the PYTHONPATH?
-        self._jsc.sc().clearFiles()
-
     def addPyFile(self, path):
         """
         Add a .py or .zip dependency for all tasks to be executed on this


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message