spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lix...@apache.org
Subject spark git commit: [SPARK-20689][PYSPARK] python doctest leaking bucketed table
Date Wed, 10 May 2017 16:33:55 GMT
Repository: spark
Updated Branches:
  refs/heads/master 5c2c4dcce -> af8b6cc82


[SPARK-20689][PYSPARK] python doctest leaking bucketed table

## What changes were proposed in this pull request?

It turns out pyspark doctest is calling saveAsTable without ever dropping them. Since we have
separate python tests for bucketed table, and there is no checking of results, there is really
no need to run the doctest, other than leaving it as an example in the generated doc

## How was this patch tested?

Jenkins

Author: Felix Cheung <felixcheung_m@hotmail.com>

Closes #17932 from felixcheung/pytablecleanup.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/af8b6cc8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/af8b6cc8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/af8b6cc8

Branch: refs/heads/master
Commit: af8b6cc82336437a55ff7578c6505d251dfa30a9
Parents: 5c2c4dc
Author: Felix Cheung <felixcheung_m@hotmail.com>
Authored: Wed May 10 09:33:49 2017 -0700
Committer: Xiao Li <gatorsmile@gmail.com>
Committed: Wed May 10 09:33:49 2017 -0700

----------------------------------------------------------------------
 python/pyspark/sql/readwriter.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/af8b6cc8/python/pyspark/sql/readwriter.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index 90ce8f8..61a6b76 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -575,7 +575,7 @@ class DataFrameWriter(OptionUtils):
         .. note:: Applicable for file-based data sources in combination with
                   :py:meth:`DataFrameWriter.saveAsTable`.
 
-        >>> (df.write.format('parquet')
+        >>> (df.write.format('parquet')  # doctest: +SKIP
         ...     .bucketBy(100, 'year', 'month')
         ...     .mode("overwrite")
         ...     .saveAsTable('bucketed_table'))
@@ -602,7 +602,7 @@ class DataFrameWriter(OptionUtils):
         :param col: a name of a column, or a list of names.
         :param cols: additional names (optional). If `col` is a list it should be empty.
 
-        >>> (df.write.format('parquet')
+        >>> (df.write.format('parquet')  # doctest: +SKIP
         ...     .bucketBy(100, 'year', 'month')
         ...     .sortBy('day')
         ...     .mode("overwrite")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message