spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ohad Raviv (JIRA)" <>
Subject [jira] [Created] (SPARK-18748) UDF multiple evaluations causes very poor performance
Date Tue, 06 Dec 2016 19:42:58 GMT
Ohad Raviv created SPARK-18748:

             Summary: UDF multiple evaluations causes very poor performance
                 Key: SPARK-18748
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.6.1
            Reporter: Ohad Raviv

We have a use case where we have a relatively expensive UDF that needs to be calculated. The
problem is that instead of being calculated once, it gets calculated over and over again.
for example:
def veryExpensiveCalc(str:String) = \{println("blahblah1"); "nothing"\}
hiveContext.udf.register("veryExpensiveCalc", veryExpensiveCalc _)
hiveContext.sql("select * from (select veryExpensiveCalc('a') c)z where c is not null and
with the output:
|      c|
You can see that for each reference of column "c" you will get the println.
that causes very poor performance for our real use case.
This also came out on StackOverflow:

with two problematic work-arounds:
1. cache() after the first time. e.g.
hiveContext.sql("select veryExpensiveCalc('a') as c").cache().where("c is not null and c<>''").show
while it works, in our case we can't do that because the table is too big to cache.

2. move back and forth to rdd:
val df = hiveContext.sql("select veryExpensiveCalc('a') as c")
hiveContext.createDataFrame(df.rdd, df.schema).where("c is not null and c<>''").show
which works but then we loose some of the optimizations like push down predicate features,
etc. and its very ugly.

Any ideas on how we can make the UDF get calculated just once in a reasonable way?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message