spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From squito <...@git.apache.org>
Subject [GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...
Date Tue, 30 Jan 2018 18:06:39 GMT
Github user squito commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20408#discussion_r164824439
  
    --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
    @@ -594,12 +606,24 @@ private[spark] class AppStatusListener(
     
           stage.executorSummaries.values.foreach(update(_, now))
           update(stage, now, last = true)
    +
    +      val executorIdsForStage = stage.executorSummaries.keySet
    +      executorIdsForStage.foreach { executorId =>
    +        liveExecutors.get(executorId).foreach { exec =>
    +          removeBlackListedStageFrom(exec, event.stageInfo.stageId, now)
    +        }
    +      }
         }
     
         appSummary = new AppSummary(appSummary.numCompletedJobs, appSummary.numCompletedStages
+ 1)
         kvstore.write(appSummary)
       }
     
    +  private def removeBlackListedStageFrom(exec: LiveExecutor, stageId: Int, now: Long)
= {
    +    exec.blacklistedInStages -= stageId
    +    liveUpdate(exec, now)
    --- End diff --
    
    hmm actually I just thought of something else.  It looks like you're calling `liveUpdate`
here for *every* executor when the stage finishes.  Say you have 1000 execs, a very quick
stage, and no blacklisting, this is an expensive update for no actual change.
    
    So you should at least avoid the `liveUpdate` if `exec.blacklistedInStages` hasn't changed
at all.  But really, I think that `LiveStage` should maintain a set of blacklisted executors,
so you avoid calling this entirely for execs which aren't blacklisted.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message