spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From viirya <...@git.apache.org>
Subject [GitHub] spark pull request #22573: [SPARK-25558][SQL] Pushdown predicates for nested...
Date Fri, 28 Sep 2018 02:54:51 GMT
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22573#discussion_r221130276
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
    @@ -437,53 +436,65 @@ object DataSourceStrategy {
        * @return a `Some[Filter]` if the input [[Expression]] is convertible, otherwise a
`None`.
        */
       protected[sql] def translateFilter(predicate: Expression): Option[Filter] = {
    +    // Recursively try to find an attribute name from the top level that can be pushed
down.
    +    def attrName(e: Expression): Option[String] = e match {
    +      // In Spark and many data sources such as parquet, dots are used as a column path
delimiter;
    +      // thus, we don't translate such expressions.
    +      case a: Attribute if !a.name.contains(".") =>
    +        Some(a.name)
    --- End diff --
    
    JDBC data source seems having no such restrict. So I worry that this change can cause
some regressions.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message