spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <>
Subject [jira] [Updated] (SPARK-20396) Add support for pandas udf in pyspark
Date Mon, 24 Apr 2017 07:52:04 GMT


Hyukjin Kwon updated SPARK-20396:
    Component/s: PySpark

> Add support for pandas udf in pyspark
> -------------------------------------
>                 Key: SPARK-20396
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: PySpark, SQL
>    Affects Versions: 2.1.0
>            Reporter: Li Jin
> split-apply-merge is a common pattern when analyzing data. It is implemented in many
popular data analyzing libraries such as Spark, Pandas, R, and etc. Split and merge operations
in these libraries are similar to each other, mostly implemented by certain grouping operators.
For instance, Spark DataFrame has groupBy, Pandas DataFrame has groupby. Therefore, for users
familiar with either Spark DataFrame or pandas DataFrame, it is not difficult for them to
understand how grouping works in the other library. However, apply is more native to different
libraries and therefore, quite different between libraries. A pandas user knows how to use
apply to do curtain transformation in pandas might not know how to do the same using pyspark.
Also, the current implementation of passing data from the java executor to python executor
is not efficient, there is opportunity to speed it up using Apache Arrow. This feature can
enable use cases that uses Spark's grouping operators such as groupBy, rollUp, cube, window
and Pandas's native apply operator.
> Related work:
> SPARK-13534
> This enables faster data serialization between Pyspark and Pandas using Apache Arrow.
Our work will be on top of this and use the same serialization for pandas udf.
> SPARK-12919 and SPARK-12922
> These implemented two functions: dapply and gapply in Spark R which implements the similar
split-apply-merge pattern that we want to implement with Pyspark. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message