spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maciej Szymkiewicz (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-19160) UDF creation
Date Tue, 10 Jan 2017 20:31:58 GMT
Maciej Szymkiewicz created SPARK-19160:
------------------------------------------

             Summary: UDF creation
                 Key: SPARK-19160
                 URL: https://issues.apache.org/jira/browse/SPARK-19160
             Project: Spark
          Issue Type: Sub-task
          Components: PySpark, SQL
    Affects Versions: 2.1.0, 2.0.0, 1.6.0, 1.5.0
            Reporter: Maciej Szymkiewicz


Right now there are a few ways we can create UDF:

- With standalone function:
{code}
def _add_one(x):
    """Adds one"""
    if x is not None:
         return x + 1            

add_one = udf(_add_one, IntegerType())
{code}
    This allows for full control flow, including exception handling, but duplicates variables.
    

- With `lambda` expression:
{code}
add_one = udf(lambda x: x + 1 if x is not None else None, IntegerType())
{code}
No variable duplication but only pure expressions.

- Using nested functions with immediate call:
{code}
def add_one(c):
    def add_one_(x):
        if x is not None:
            return x + 1
    return udf(add_one_, IntegerType())(c)
{code}
Quite verbose but enables full control flow and clearly indicates expected number of arguments.
    
- Using `udf` functions as a decorator:
{code}
@udf
def add_one(x):
    """Adds one"""
    if x is not None:
        return x + 1
{code}
Possible but only with default `returnType` (or curried `@partial(udf, returnType=IntegerType())`).
    
Proposed

Add `udf` decorator which can be used as follows:

{code}
from pyspark.sql.decorators import udf

@udf(IntegerType())
def add_one(x):
    """Adds one"""
    if x is not None:
        return x + 1
{code}

or 

{code}
@udf()
def strip(x):
    """Strips String"""
    if x is not None:
        return x.strip()
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message