spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Jin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-22239) Used-defined window functions with pandas udf
Date Tue, 10 Oct 2017 22:54:00 GMT
Li Jin created SPARK-22239:
------------------------------

             Summary: Used-defined window functions with pandas udf
                 Key: SPARK-22239
                 URL: https://issues.apache.org/jira/browse/SPARK-22239
             Project: Spark
          Issue Type: Sub-task
          Components: PySpark
    Affects Versions: 2.2.0
         Environment: 

            Reporter: Li Jin


Window function is another place we can benefit from vectored udf and add another useful function
to the pandas_udf suite.

Example usage (preliminary):

{code:java}
w = Window.partitionBy('id').orderBy('time').rangeBetween(-200, 0)

@pandas_udf(DoubleType())
def ema(v1):
    return v1.ewm(alpha=0.5).mean().iloc[-1]

df.withColumn('v1_ema', ema(df.v1).over(window))
{code}





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message