spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From szhem <...@git.apache.org>
Subject [GitHub] spark pull request #19294: [SPARK-21549][CORE] Respect OutputFormats with no...
Date Tue, 03 Oct 2017 06:09:19 GMT
Github user szhem commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19294#discussion_r142319273
  
    --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
---
    @@ -57,6 +57,15 @@ class HadoopMapReduceCommitProtocol(jobId: String, path: String)
        */
       private def absPathStagingDir: Path = new Path(path, "_temporary-" + jobId)
     
    +  /**
    +   * Checks whether there are files to be committed to an absolute output location.
    +   *
    +   * As the committing and aborting the job occurs on driver where `addedAbsPathFiles`
is always
    +   * null, it is necessary to check whether the output path is specified, that may not
be the case
    +   * for committers not writing to distributed file systems.
    +   */
    +  private def hasAbsPathFiles: Boolean = path != null
    --- End diff --
    
    done


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message