spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gurwls...@apache.org
Subject spark git commit: [SPARK-24444][DOCS][PYTHON] Improve Pandas UDF docs to explain column assignment
Date Fri, 01 Jun 2018 03:59:05 GMT
Repository: spark
Updated Branches:
  refs/heads/master cbaa72913 -> b2d022656


[SPARK-24444][DOCS][PYTHON] Improve Pandas UDF docs to explain column assignment

## What changes were proposed in this pull request?

Added sections to pandas_udf docs, in the grouped map section, to indicate columns are assigned
by position.

## How was this patch tested?

NA

Author: Bryan Cutler <cutlerb@gmail.com>

Closes #21471 from BryanCutler/arrow-doc-pandas_udf-column_by_pos-SPARK-21427.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b2d02265
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b2d02265
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b2d02265

Branch: refs/heads/master
Commit: b2d022656298c7a39ff3e84b04f813d5f315cb95
Parents: cbaa729
Author: Bryan Cutler <cutlerb@gmail.com>
Authored: Fri Jun 1 11:58:59 2018 +0800
Committer: hyukjinkwon <gurwls223@apache.org>
Committed: Fri Jun 1 11:58:59 2018 +0800

----------------------------------------------------------------------
 docs/sql-programming-guide.md   | 9 +++++++++
 python/pyspark/sql/functions.py | 9 ++++++++-
 2 files changed, 17 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/b2d02265/docs/sql-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md
index 5060086..4d8a738 100644
--- a/docs/sql-programming-guide.md
+++ b/docs/sql-programming-guide.md
@@ -1752,6 +1752,15 @@ To use `groupBy().apply()`, the user needs to define the following:
 * A Python function that defines the computation for each group.
 * A `StructType` object or a string that defines the schema of the output `DataFrame`.
 
+The output schema will be applied to the columns of the returned `pandas.DataFrame` in order
by position,
+not by name. This means that the columns in the `pandas.DataFrame` must be indexed so that
their
+position matches the corresponding field in the schema.
+
+Note that when creating a new `pandas.DataFrame` using a dictionary, the actual position
of the column
+can differ from the order that it was placed in the dictionary. It is recommended in this
case to
+explicitly define the column order using the `columns` keyword, e.g.
+`pandas.DataFrame({'id': ids, 'a': data}, columns=['id', 'a'])`, or alternatively use an
`OrderedDict`.
+
 Note that all data for a group will be loaded into memory before the function is applied.
This can
 lead to out of memory exceptons, especially if the group sizes are skewed. The configuration
for
 [maxRecordsPerBatch](#setting-arrow-batch-size) is not applied on groups and it is up to
the user

http://git-wip-us.apache.org/repos/asf/spark/blob/b2d02265/python/pyspark/sql/functions.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index efcce25..fd656c5 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -2500,7 +2500,8 @@ def pandas_udf(f=None, returnType=None, functionType=None):
        A grouped map UDF defines transformation: A `pandas.DataFrame` -> A `pandas.DataFrame`
        The returnType should be a :class:`StructType` describing the schema of the returned
        `pandas.DataFrame`.
-       The length of the returned `pandas.DataFrame` can be arbitrary.
+       The length of the returned `pandas.DataFrame` can be arbitrary and the columns must
be
+       indexed so that their position matches the corresponding field in the schema.
 
        Grouped map UDFs are used with :meth:`pyspark.sql.GroupedData.apply`.
 
@@ -2548,6 +2549,12 @@ def pandas_udf(f=None, returnType=None, functionType=None):
        |  2|6.0|
        +---+---+
 
+       .. note:: If returning a new `pandas.DataFrame` constructed with a dictionary, it
is
+           recommended to explicitly index the columns by name to ensure the positions are
correct,
+           or alternatively use an `OrderedDict`.
+           For example, `pd.DataFrame({'id': ids, 'a': data}, columns=['id', 'a'])` or
+           `pd.DataFrame(OrderedDict([('id', ids), ('a', data)]))`.
+
        .. seealso:: :meth:`pyspark.sql.GroupedData.apply`
 
     3. GROUPED_AGG


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message