spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Jin (JIRA)" <>
Subject [jira] [Created] (SPARK-22239) Used-defined window functions with pandas udf
Date Tue, 10 Oct 2017 22:54:00 GMT
Li Jin created SPARK-22239:

             Summary: Used-defined window functions with pandas udf
                 Key: SPARK-22239
             Project: Spark
          Issue Type: Sub-task
          Components: PySpark
    Affects Versions: 2.2.0

            Reporter: Li Jin

Window function is another place we can benefit from vectored udf and add another useful function
to the pandas_udf suite.

Example usage (preliminary):

w = Window.partitionBy('id').orderBy('time').rangeBetween(-200, 0)

def ema(v1):
    return v1.ewm(alpha=0.5).mean().iloc[-1]

df.withColumn('v1_ema', ema(df.v1).over(window))

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message