spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From saucam <>
Subject [GitHub] spark pull request: SPARK-3968 Use parquet-mr filter2 api in spark...
Date Mon, 27 Oct 2014 15:12:20 GMT
Github user saucam commented on a diff in the pull request:
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala
    @@ -450,8 +465,14 @@ private[parquet] class FilteringParquetRowInputFormat
    +    val filter: Filter = ParquetInputFormat.getFilter(configuration)
    +    var rowGroupsDropped :Long = 0
    +    var totalRowGroups :Long  = 0
    +    // Ugly hack, stuck with it until mentioned PR is resolved
         val generateSplits =
    -      classOf[ParquetInputFormat[_]].getDeclaredMethods.find(_.getName == "generateSplits").get
    +      Class.forName("parquet.hadoop.ClientSideMetadataSplitStrategy")
    --- End diff --
    Thanks for pointing this out, I had chosen ClientSideMetadataStrategy because it was the
default one (had left the other as TODO), in fact we indeed should use  TaskSideMetadataSplitStrategy
. For now I have included the code for both strategies, and made TaskSideMetadataSplitStrategy
default. Should we create a config for this ?

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message