spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lix...@apache.org
Subject spark git commit: [SPARK-23624][SQL] Revise doc of method pushFilters in Datasource V2
Date Fri, 09 Mar 2018 23:41:23 GMT
Repository: spark
Updated Branches:
  refs/heads/master 2ca9bb083 -> 10b0657b0


[SPARK-23624][SQL] Revise doc of method pushFilters in Datasource V2

## What changes were proposed in this pull request?

Revise doc of method pushFilters in SupportsPushDownFilters/SupportsPushDownCatalystFilters

In `FileSourceStrategy`, except `partitionKeyFilters`(the references of which is subset of
partition keys), all filters needs to be evaluated after scanning. Otherwise, Spark will get
wrong result from data sources like Orc/Parquet.

This PR is to improve the doc.

Author: Wang Gengliang <gengliang.wang@databricks.com>

Closes #20769 from gengliangwang/revise_pushdown_doc.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/10b0657b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/10b0657b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/10b0657b

Branch: refs/heads/master
Commit: 10b0657b035641ce735055bba2c8459e71bc2400
Parents: 2ca9bb0
Author: Wang Gengliang <gengliang.wang@databricks.com>
Authored: Fri Mar 9 15:41:19 2018 -0800
Committer: gatorsmile <gatorsmile@gmail.com>
Committed: Fri Mar 9 15:41:19 2018 -0800

----------------------------------------------------------------------
 .../sql/sources/v2/reader/SupportsPushDownCatalystFilters.java     | 2 +-
 .../spark/sql/sources/v2/reader/SupportsPushDownFilters.java       | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/10b0657b/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownCatalystFilters.java
----------------------------------------------------------------------
diff --git a/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownCatalystFilters.java
b/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownCatalystFilters.java
index 9822410..290d614 100644
--- a/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownCatalystFilters.java
+++ b/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownCatalystFilters.java
@@ -34,7 +34,7 @@ import org.apache.spark.sql.catalyst.expressions.Expression;
 public interface SupportsPushDownCatalystFilters extends DataSourceReader {
 
   /**
-   * Pushes down filters, and returns unsupported filters.
+   * Pushes down filters, and returns filters that need to be evaluated after scanning.
    */
   Expression[] pushCatalystFilters(Expression[] filters);
 

http://git-wip-us.apache.org/repos/asf/spark/blob/10b0657b/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownFilters.java
----------------------------------------------------------------------
diff --git a/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownFilters.java
b/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownFilters.java
index f35c711..1cff024 100644
--- a/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownFilters.java
+++ b/sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsPushDownFilters.java
@@ -32,7 +32,7 @@ import org.apache.spark.sql.sources.Filter;
 public interface SupportsPushDownFilters extends DataSourceReader {
 
   /**
-   * Pushes down filters, and returns unsupported filters.
+   * Pushes down filters, and returns filters that need to be evaluated after scanning.
    */
   Filter[] pushFilters(Filter[] filters);
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message