spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang-Chi Hsieh (JIRA)" <>
Subject [jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions
Date Mon, 18 Apr 2016 10:21:25 GMT


Liang-Chi Hsieh commented on SPARK-14083:

Based on [~joshrosen]'s code, I added some comments and few Java opcodes:

> Analyze JVM bytecode and turn closures into Catalyst expressions
> ----------------------------------------------------------------
>                 Key: SPARK-14083
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>            Reporter: Reynold Xin
> One big advantage of the Dataset API is the type safety, at the cost of performance due
to heavy reliance on user-defined closures/lambdas. These closures are typically slower than
expressions because we have more flexibility to optimize expressions (known data types, no
virtual function calls, etc). In many cases, it's actually not going to be very difficult
to look into the byte code of these closures and figure out what they are trying to do. If
we can understand them, then we can turn them directly into Catalyst expressions for more
optimized executions.
> Some examples are:
> {code}
>  // equivalent to expression col("name")
> ds.groupBy(_.gender)  // equivalent to expression col("gender")
> df.filter(_.age > 18)  // equivalent to expression GreaterThan(col("age"), lit(18)
> + 1)  // equivalent to Add(col("age"), lit(1))
> {code}
> The goal of this ticket is to design a small framework for byte code analysis and use
that to convert closures/lambdas into Catalyst expressions in order to speed up Dataset execution.
It is a little bit futuristic, but I believe it is very doable. The framework should be easy
to reason about (e.g. similar to Catalyst).
> Note that a big emphasis on "small" and "easy to reason about". A patch should be rejected
if it is too complicated or difficult to reason about.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message