spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hvanhovell <>
Subject [GitHub] spark pull request: [SPARK-7712] [SQL] Move Window Functions from ...
Date Wed, 20 May 2015 07:54:04 GMT
GitHub user hvanhovell opened a pull request:

    [SPARK-7712] [SQL] Move Window Functions from Hive UDAFS to Spark Native backend [WIP]

    This PR aims to improve the current window functionality in Spark SQL in the following
    - Moving away from Hive backed UDAF's to a more Spark SQL Native implementation. The main
advantages are that some overhead can be avoided, that it it easier to extend, and that it
opens up the opportunity to do more aggressive code generation (AggregateEvaluation style).
    - Improve the processing for the 'between unbounded and current' amd the 'between current
and unbounded' cases. This was very expensive, now it performs in linear time.
    - Process different frames for the same group by/order by definition in the same window
clause. This should save some memory and reduce some of the overhead.
    This is currently a work in progress. Quite a bit of testing needs to be done. Any feedback
will be greatly appreciated. The JIRA ticket can be found here:

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-7712

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #6278
commit e4b81d5276076ce1f8130b56c557ff9a4e19dce8
Author: Herman van Hovell <>
Date:   2015-05-20T07:30:38Z

    Moved Code from separate project into Spark code base. Added HiveQL parsing.


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message