spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From liancheng <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-8756][SQL] Keep cached information and ...
Date Wed, 22 Jul 2015 10:18:34 GMT
Github user liancheng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/7154#discussion_r35198847
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala ---
    @@ -345,14 +345,23 @@ private[sql] class ParquetRelation2(
         // Schema of the whole table, including partition columns.
         var schema: StructType = _
     
    +    // Cached leaves
    +    val cachedLeaves: mutable.Map[Path, FileStatus] = new mutable.HashMap[Path, FileStatus]()
    +
         /**
          * Refreshes `FileStatus`es, footers, partition spec, and table schema.
          */
         def refresh(): Unit = {
           // Lists `FileStatus`es of all leaf nodes (files) under all base directories.
    +      // We only care about new FileStatus and updated FileStatus.
           val leaves = cachedLeafStatuses().filter { f =>
    -        isSummaryFile(f.getPath) ||
    -          !(f.getPath.getName.startsWith("_") || f.getPath.getName.startsWith("."))
    +        (isSummaryFile(f.getPath) ||
    +          !(f.getPath.getName.startsWith("_") || f.getPath.getName.startsWith(".")))
&&
    +          (!cachedLeaves.contains(f.getPath) ||
    +            cachedLeaves(f.getPath).getModificationTime < f.getModificationTime)
    +      }.map { f =>
    +        cachedLeaves += (f.getPath -> f)
    +        f
    --- End diff --
    
    I guess you are trying to only keep `FileStatus`es of those files that need to be touched
during schema merging here. Actually the `FileStatus` cache must be consistent with the files
stored on the file system. Because we also inject the cache to `ParquetInputFormat` in `ParquetRelation2.buildScan`
to avoid calling `listStatus` repeatedly there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message