spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HyukjinKwon <...@git.apache.org>
Subject [GitHub] spark pull request #20830: [SPARK-23691][PYTHON] Use sql_conf util in PySpar...
Date Thu, 15 Mar 2018 04:24:42 GMT
GitHub user HyukjinKwon opened a pull request:

    https://github.com/apache/spark/pull/20830

    [SPARK-23691][PYTHON] Use sql_conf util in PySpark tests where possible

    ## What changes were proposed in this pull request?
    
    https://github.com/apache/spark/commit/d6632d185e147fcbe6724545488ad80dce20277e added
an useful util
    
    ```python
    @contextmanager
    def sql_conf(self, pairs):
        ...
    ```
    
    to allow configuration set/unset within a block:
    
    ```python
    with self.sql_conf({"spark.blah.blah.blah", "blah"})
        # test codes
    ```
    
    This PR proposes to use this util where possible in PySpark tests.
    
    Note that there look already few places affecting tests without restoring the original
value back in unittest classes.
    
    ## How was this patch tested?
    
    Manually tested via:
    
    ```
    ./run-tests --modules=pyspark-sql --python-executables=python2
    ./run-tests --modules=pyspark-sql --python-executables=python3
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HyukjinKwon/spark cleanup-sql-conf

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20830.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20830
    
----
commit 89cf69be7ae00f571c51de402067928b663c5a45
Author: hyukjinkwon <gurwls223@...>
Date:   2018-03-15T04:16:18Z

    Use sql_conf util in PySpark tests where possible

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message