spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cloud-fan <...@git.apache.org>
Subject [GitHub] spark pull request #21320: [SPARK-4502][SQL] Parquet nested column pruning -...
Date Mon, 02 Jul 2018 15:36:47 GMT
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21320#discussion_r199516625
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
    @@ -1288,8 +1288,18 @@ object SQLConf {
             "issues. Turn on this config to insert a local sort before actually doing repartition
" +
             "to generate consistent repartition results. The performance of repartition()
may go " +
             "down since we insert extra local sort before it.")
    +        .booleanConf
    +        .createWithDefault(true)
    +
    +  val NESTED_SCHEMA_PRUNING_ENABLED =
    +    buildConf("spark.sql.nestedSchemaPruning.enabled")
    +      .internal()
    +      .doc("Prune nested fields from a logical relation's output which are unnecessary
in " +
    +        "satisfying a query. This optimization allows columnar file format readers to
avoid " +
    +        "reading unnecessary nested column data. Currently Parquet is the only data source
that " +
    +        "implements this optimization.")
           .booleanConf
    -      .createWithDefault(true)
    +      .createWithDefault(false)
    --- End diff --
    
    +1


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message