spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cloud-fan <...@git.apache.org>
Subject [GitHub] spark pull request #21004: [SPARK-23896][SQL]Improve PartitioningAwareFileIn...
Date Fri, 13 Apr 2018 02:04:39 GMT
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21004#discussion_r181269578
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
    @@ -552,6 +523,40 @@ case class DataSource(
             sys.error(s"${providingClass.getCanonicalName} does not allow create table as
select.")
         }
       }
    +
    +  /** Returns an [[InMemoryFileIndex]] that can be used to get partition schema and file
list. */
    +  private def createInMemoryFileIndex(globbedPaths: Seq[Path]): InMemoryFileIndex = {
    --- End diff --
    
    and we can merge `checkAndGlobPathIfNecessary` and `createInMemoryFileIndex`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message