spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jiangxb1987 <...@git.apache.org>
Subject [GitHub] spark pull request #19388: [SPARK-22162] Executors and the driver should use...
Date Mon, 02 Oct 2017 15:01:30 GMT
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19388#discussion_r142163204
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
    @@ -864,6 +873,45 @@ class NewFakeFormatWithCallback() extends NewFakeFormat {
       }
     }
     
    +class YetAnotherFakeCommitter extends NewOutputCommitter with Assertions {
    +  def setupJob(j: NewJobContext): Unit = {
    +    JobID.jobid = j.getJobID().getId
    +  }
    +
    +  def needsTaskCommit(t: NewTaskAttempContext): Boolean = false
    +
    +  def setupTask(t: NewTaskAttempContext): Unit = {
    +    val rddid = t.getTaskAttemptID().getJobID().getId
    --- End diff --
    
    You don't need to ensure the stageId is equal to `rddId`, just make sure it inline with
the jobId is okay.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message