spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jianshi Huang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-6382) withUDF(...) {...} for supporting temporary UDF definitions in the scope
Date Tue, 17 Mar 2015 09:20:38 GMT
Jianshi Huang created SPARK-6382:
------------------------------------

             Summary: withUDF(...) {...} for supporting temporary UDF definitions in the scope
                 Key: SPARK-6382
                 URL: https://issues.apache.org/jira/browse/SPARK-6382
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 1.3.0, 1.3.1
            Reporter: Jianshi Huang


Currently the scope of UDF registration is global. It's unsuitable for libraries that's built
on top of DataFrame, as many operations has to done by registering a UDF first.

Please provide a way for binding temporary UDFs.

e.g.

{code}
withUDF(("merge_map", (m1: Map[String, Double], m2: Map[String, Double]) => m2 ++ m2),
    ...) {
  sql("select merge_map(d1.map, d2.map) from d1, d2 where d1.id = d2.id")
}
{code}

Also UDF registry is a mutable Hashmap, refactoring it to a immutable one makes more sense.

Jianshi




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message