spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jiangxb1987 <...@git.apache.org>
Subject [GitHub] spark pull request #20931: [SPARK-23815][Core]Spark writer dynamic partition...
Date Fri, 06 Apr 2018 03:07:09 GMT
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20931#discussion_r179651037
  
    --- Diff: core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
---
    @@ -186,7 +186,9 @@ class HadoopMapReduceCommitProtocol(
             logDebug(s"Clean up default partition directories for overwriting: $partitionPaths")
             for (part <- partitionPaths) {
               val finalPartPath = new Path(path, part)
    -          fs.delete(finalPartPath, true)
    +          if (!fs.delete(finalPartPath, true) && !fs.exists(finalPartPath.getParent))
{
    --- End diff --
    
    I think the problem here is we didn't check whether the `finalPartPath` exists, and we
shall actually check that before rename. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message