spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "xdcjie (JIRA)" <>
Subject [jira] [Created] (SPARK-19269) Scheduler Delay in spark ui
Date Wed, 18 Jan 2017 06:14:26 GMT
xdcjie created SPARK-19269:

             Summary: Scheduler Delay in spark ui
                 Key: SPARK-19269
             Project: Spark
          Issue Type: Question
          Components: Web UI
    Affects Versions: 2.0.2
            Reporter: xdcjie

In sparkUI, SchedulerDelay is launchtime(in driver) - starttime (in executor), but in the
code, that it consists of launchtime - starttime and statupdate (executor update status to
driver time).
private[ui] def getSchedulerDelay(
      info: TaskInfo, metrics: TaskMetricsUIData, currentTime: Long): Long = {
    if (info.finished) {
      val totalExecutionTime = info.finishTime - info.launchTime
      val executorOverhead = (metrics.executorDeserializeTime +
        totalExecutionTime - metrics.executorRunTime - executorOverhead -
          getGettingResultTime(info, currentTime))
    } else {
      // The task is still running and the metrics like executorRunTime are not available.

Could we split "scheduler delay" into "scheduler delay time (launchtime-starttime) and "status
update time".  in my workspace, scheduler delay time is 60s,   but launchtime - starttime
is very fast, <1s.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message