spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: [DAGScheduler] resubmitFailedStages, failedStages.clear() and submitStage
Date Thu, 24 Dec 2015 15:27:15 GMT
getMissingParentStages(stage) would be called for the stage (being
re-submitted)

If there is no missing parents, submitMissingTasks() would be called.
If there is missing parent(s), the parent would go through the same flow.

I don't see issue in this part of the code.

Cheers

On Thu, Dec 24, 2015 at 5:19 AM, Jacek Laskowski <jacek@japila.pl> wrote:

> Hi,
>
> While reviewing DAGScheduler, and where failedStages internal
> collection of failed staged ready for resubmission is used, I came
> across a question for which I'm looking an answer to. Any hints would
> be greatly appreciated.
>
> When resubmitFailedStages [1] is executed, and there are any failed
> stages, they are resubmitted using submitStage [2], but before it
> happens, failedStages is cleared [3] so when submitStage is called
> that will ultimately call submitMissingTasks for the stage, it checks
> whether the stage is in failedStages (among the other sets for waiting
> and running stages) [4].
>
> My naive understanding is that the call to submitStage is a no-op in
> this case, i.e. nothing really happens and the if expression will
> silently finish without doing anything useful until some other event
> happens that changes the status of the failed stages into waiting
> ones.
>
> Is my understanding incorrect? Where? Could the call to submitStage be
> superfluous? Please guide in the right direction. Thanks.
>
> [1]
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L734
> [2]
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L743
> [3]
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L741
> [4]
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L919
>
> Pozdrawiam,
> Jacek
>
> Jacek Laskowski | https://medium.com/@jaceklaskowski/
> Mastering Apache Spark
> ==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message