spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yhuai <>
Subject [GitHub] spark pull request: [SPARK-10539][SQL]Project should not be pushed...
Date Fri, 18 Sep 2015 18:14:40 GMT
GitHub user yhuai opened a pull request:

    [SPARK-10539][SQL]Project should not be pushed down through Intersect or Except #8742

    Intersect and Except are both set operators and they use the all the columns to compare
equality between rows. When pushing their Project parent down, the relations they based on
would change, therefore not an equivalent transformation.
    I added some comments based on the fix of

You can merge this pull request into a Git repository by running:

    $ git pull fix_set_optimization

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #8823
commit a9aa64388bae6afffecb27c432d9cf3de77aeba0
Author: Yijie Shen <>
Date:   2015-09-14T09:05:38Z

    fix set optimization by eliminate empty project push down

commit f9a3b70c7f4c73c7f0572c31a90bb7f7b4698ef7
Author: Yijie Shen <>
Date:   2015-09-14T16:30:38Z

    Project should not be pushed down through Intersect or Except

commit 1f56d2e626ed75d4ce96ffd70929243eed4c8143
Author: Yin Huai <>
Date:   2015-09-18T18:13:05Z

    Add comments.


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message