spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ueshin <...@git.apache.org>
Subject [GitHub] spark pull request #20515: [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use date...
Date Tue, 06 Feb 2018 07:47:27 GMT
GitHub user ueshin opened a pull request:

    https://github.com/apache/spark/pull/20515

    [SPARK-23290][SQL][PYTHON][BACKPORT-2.3] Use datetime.date for date type when converting
Spark DataFrame to Pandas DataFrame.

    ## What changes were proposed in this pull request?
    
    This is a backport of #20506.
    
    In #18664, there was a change in how `DateType` is being returned to users ([line 1968
in dataframe.py](https://github.com/apache/spark/pull/18664/files#diff-6fc344560230bf0ef711bb9b5573f1faR1968)).
This can cause client code which works in Spark 2.2 to fail.
    See [SPARK-23290](https://issues.apache.org/jira/browse/SPARK-23290?focusedCommentId=16350917&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16350917)
for an example.
    
    This pr modifies to use `datetime.date` for date type as Spark 2.2 does.
    
    ## How was this patch tested?
    
    Tests modified to fit the new behavior and existing tests.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ueshin/apache-spark issues/SPARK-23290_2.3

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20515.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20515
    
----
commit b489f4a0d4fa25fd51d9db78bd01fc972e4e0dd4
Author: Takuya UESHIN <ueshin@...>
Date:   2018-02-06T06:52:25Z

    [SPARK-23290][SQL][PYTHON] Use datetime.date for date type when converting Spark DataFrame
to Pandas DataFrame.
    
    ## What changes were proposed in this pull request?
    
    In #18664, there was a change in how `DateType` is being returned to users ([line 1968
in dataframe.py](https://github.com/apache/spark/pull/18664/files#diff-6fc344560230bf0ef711bb9b5573f1faR1968)).
This can cause client code which works in Spark 2.2 to fail.
    See [SPARK-23290](https://issues.apache.org/jira/browse/SPARK-23290?focusedCommentId=16350917&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16350917)
for an example.
    
    This pr modifies to use `datetime.date` for date type as Spark 2.2 does.
    
    ## How was this patch tested?
    
    Tests modified to fit the new behavior and existing tests.
    
    Author: Takuya UESHIN <ueshin@databricks.com>
    
    Closes #20506 from ueshin/issues/SPARK-23290.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message