spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HyukjinKwon <>
Subject [GitHub] spark pull request: [SPARK-11676][SQL] Parquet filter tests all pa...
Date Thu, 12 Nov 2015 09:50:31 GMT
GitHub user HyukjinKwon opened a pull request:

    [SPARK-11676][SQL] Parquet filter tests all pass if filters are not really pushed down

    Currently Parquet predicate tests all pass even if filters are not pushed down or this
is disabled.
    In this PR, For checking evaluating filters, Simply it makes the expression from `expression.Filter`
and then try to create filters just like Spark does.
    For checking the results, this manually accesses to the child rdd (of `expression.Filter`)
and produces the results which should be filtered properly, and then compares it to expected
    Now, if filters are not pushed down or this is disabled, this throws exceptions.
    cc @liancheng 

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-11676

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #9659
commit 1c6a5480df53d16fe38d14146deedaf035a6024c
Author: hyukjinkwon <>
Date:   2015-11-12T09:31:41Z

    [SPARK-11676][SQL] Parquet filter tests all pass if filters are not really pushed down

commit 6b722ef4ad3a29a1bd0cde0961f6bc12638e2f31
Author: hyukjinkwon <>
Date:   2015-11-12T09:37:27Z

    [SPARK-11676][SQL] Update indentation and comments

commit d007c3f996f1fd4e093a590741b92a4bc4072d41
Author: hyukjinkwon <>
Date:   2015-11-12T09:39:05Z

    [SPARK-11676][SQL] Remove unused imports

commit 8cc842fa3f97c39e61fa072a916f7056a85be1b7
Author: hyukjinkwon <>
Date:   2015-11-12T09:49:11Z

    [SPARK-111676][SQL] Set true to pushing down filters


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message