spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rxin <>
Subject [GitHub] spark pull request #18299: Spark 21092
Date Wed, 14 Jun 2017 07:24:37 GMT
GitHub user rxin opened a pull request:

    Spark 21092

    ## What changes were proposed in this pull request?
    It is really painful to not have configs in logical plan and expressions. We had to add
all sorts of hacks (e.g. pass SQLConf explicitly in functions). This patch exposes SQLConf
in logical plan, using a thread local variable and a getter closure that's set once there
is an active SparkSession.
    The implementation is a bit of a hack, since we didn't anticipate this need in the beginning
(config was only exposed in physical plan).
    In terms of future work, we should follow up to clean up CBO (remove the need for passing
in config).
    ## How was this patch tested?
    Updated relevant tests for constraint propagation.

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-21092

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18299
commit bbcd26f0cdfaba602eacbcac79a563e06cb33736
Author: Reynold Xin <>
Date:   2017-06-14T06:41:29Z

    [SPARK-21091][SQL] Move constraint code into QueryPlanConstraints

commit ced6e9b754fee8df23b0b554a621c185c2a2b211
Author: Reynold Xin <>
Date:   2017-06-14T07:19:13Z

    [SPARK-21092][SQL] Wire SQLConf in logical plan and expressions


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message