spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kris Mok (JIRA)" <>
Subject [jira] [Commented] (SPARK-25140) Add optional logging to UnsafeProjection.create when it falls back to interpreted mode
Date Fri, 17 Aug 2018 07:16:00 GMT


Kris Mok commented on SPARK-25140:

Thanks maropu-san! No, I'm not working on this ticket; I'm currently swamped by other tasks
close to a deadline so I couldn't work on this one right now.

I was just thinking about resetting my mental model for the Spark 2.4.x code execution performance,
and saw that after the fallback was implemented, it's very hard for the user to figure out
whether or not the interpreter fallback is taking effect and/or whether or not it's contributing
to slow performance. This would be valuable information to have for query tuning, etc.

> Add optional logging to UnsafeProjection.create when it falls back to interpreted mode
> --------------------------------------------------------------------------------------
>                 Key: SPARK-25140
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: Kris Mok
>            Priority: Minor
> SPARK-23711 implemented a nice graceful handling of allowing UnsafeProjection to fall
back to an interpreter mode when codegen fails. That makes Spark much more usable even when
codegen is unable to handle the given query.
> But in its current form, the fallback handling can also be a mystery in terms of performance
cliffs. Users may be left wondering why a query runs fine with some expressions, but then
with just one extra expression the performance goes 2x, 3x (or more) slower.
> It'd be nice to have optional logging of the fallback behavior, so that for users that
care about monitoring performance cliffs, they can opt-in to log when a fallback to interpreter
mode was taken. i.e. at

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message