spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [18/50] git commit: Merge pull request #205 from kayousterhout/logging
Date Thu, 12 Dec 2013 07:11:49 GMT
Merge pull request #205 from kayousterhout/logging

Added logging of scheduler delays to UI

This commit adds two metrics to the UI:

1) The time to get task results, if they're fetched remotely

2) The scheduler delay.  When the scheduler starts getting overwhelmed (because it can't keep
up with the rate at which tasks are being submitted), the result is that tasks get delayed
on the tail-end: the message from the worker saying that the task has completed ends up in
a long queue and takes a while to be processed by the scheduler.  This commit records that
delay in the UI so that users can tell when the scheduler is becoming the bottleneck.


Branch: refs/heads/scala-2.10
Commit: 3fb302c08d078decd1fa7dd0fc008faff132ab7f
Parents: 87676a6 58b3aff
Author: Matei Zaharia <>
Authored: Fri Dec 6 11:03:32 2013 -0800
Committer: Matei Zaharia <>
Committed: Fri Dec 6 11:03:32 2013 -0800

 .../org/apache/spark/ui/jobs/StagePage.scala    | 36 ++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)

View raw message