spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From squito <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-8103][core] DAGScheduler should not sub...
Date Wed, 15 Jul 2015 18:21:11 GMT
Github user squito commented on a diff in the pull request:

    https://github.com/apache/spark/pull/6750#discussion_r34711221
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
    @@ -319,26 +337,24 @@ private[spark] class TaskSchedulerImpl(
                 failedExecutor = Some(execId)
               }
             }
    -        taskIdToTaskSetId.get(tid) match {
    -          case Some(taskSetId) =>
    +        taskSetManagerForTask(tid) match {
    +          case Some(taskSet) =>
                 if (TaskState.isFinished(state)) {
    -              taskIdToTaskSetId.remove(tid)
    +              taskIdToStageIdAndAttempt.remove(tid)
                   taskIdToExecutorId.remove(tid)
                 }
    -            activeTaskSets.get(taskSetId).foreach { taskSet =>
    -              if (state == TaskState.FINISHED) {
    -                taskSet.removeRunningTask(tid)
    -                taskResultGetter.enqueueSuccessfulTask(taskSet, tid, serializedData)
    -              } else if (Set(TaskState.FAILED, TaskState.KILLED, TaskState.LOST).contains(state))
{
    -                taskSet.removeRunningTask(tid)
    -                taskResultGetter.enqueueFailedTask(taskSet, tid, state, serializedData)
    -              }
    +            if (state == TaskState.FINISHED) {
    +              taskSet.removeRunningTask(tid)
    +              taskResultGetter.enqueueSuccessfulTask(taskSet, tid, serializedData)
    +            } else if (Set(TaskState.FAILED, TaskState.KILLED, TaskState.LOST).contains(state))
{
    +              taskSet.removeRunningTask(tid)
    +              taskResultGetter.enqueueFailedTask(taskSet, tid, state, serializedData)
                 }
               case None =>
                 logError(
    --- End diff --
    
    there is a small change in logic here.  Before, we'd hit this case only if `taskIdToTaskSetId`
didn't contain `tid`.  If `activeTaskSets` doesn't contain `taskSetId`, we'd do nothing (foreach
on None).  But now, we hit the error case if either `taskIdToTaskSetId` doesn't contain `tid`,
or if `taskSetsByStageIdAndAttempt` doesn't contain the stage & attempt.
    
    This wasn't exactly intentional, it just fell out from an attempt at refactoring a helper
method -- but it does seems to more accurately correspond to what is in the error msg.  Still
I'm not certain about it.  Perhaps if we nobody else has high confidence in this change I
should just go back to the old behavior.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message