spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <>
Subject [jira] [Commented] (SPARK-4440) Enhance the job progress API to expose more information
Date Thu, 17 Sep 2015 07:43:45 GMT


Rui Li commented on SPARK-4440:

For Hive on Spark, we want completion time for each stage so we can compute how long the stage
takes(there's already a submission time in {{SparkStageInfo}}).
It'll be great if we can also get task metrics. Currently we have to implement SparkListener
to collect metrics.

[~chengxiang li] and [~xuefuz], do you have anything to add?

> Enhance the job progress API to expose more information
> -------------------------------------------------------
>                 Key: SPARK-4440
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Rui Li
> The progress API introduced in SPARK-2321 provides a new way for user to monitor job
progress. However the information exposed in the API is relatively limited. It'll be much
more useful if we can enhance the API to expose more data.
> Some improvement for example may include but not limited to:
> 1. Stage submission and completion time.
> 2. Task metrics.
> The requirement is initially identified for the hive on spark project(HIVE-7292), other
application should benefit as well.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message