spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anton Okolnychyi (JIRA)" <>
Subject [jira] [Updated] (SPARK-26204) Optimize InSet expression
Date Wed, 28 Nov 2018 16:34:00 GMT


Anton Okolnychyi updated SPARK-26204:
    Attachment: heap size.png

> Optimize InSet expression
> -------------------------
>                 Key: SPARK-26204
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Anton Okolnychyi
>            Priority: Major
>         Attachments: heap size.png
> The {{InSet}} expression was introduced in SPARK-3711 to avoid O\(n\) time complexity
in the `In` expression. As {{InSet}} relies on Scala {{immutable.Set}}, it introduces expensive
autoboxing. As a consequence, the performance of {{InSet}} might be significantly slower than
{{In}} even on 100+ values.
> We need to find an approach how to optimize {{InSet}} expressions and avoid the cost
of autoboxing.
>  There are a few approaches that we can use:
>  * Collections for primitive values (e.g., FastUtil,  HPPC)
>  * Type specialization in Scala (would it even work for code gen in Spark?)
> I tried to use {{OpenHashSet}}, which is already available in Spark and uses type specialization.
However, I did not manage to avoid autoboxing. On the other hand, FastUtil did work and I
saw a substantial improvement in the performance.
> See the attached screenshot of what I experienced while testing.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message