spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dev Lakhani (JIRA)" <>
Subject [jira] [Commented] (SPARK-2321) Design a proper progress reporting & event listener API
Date Thu, 16 Oct 2014 11:02:33 GMT


Dev Lakhani commented on SPARK-2321:

There are some issues and bugs under the webui component that are active. Should we incorporate
these into this Jira or is it best to work on them separately and then merge these (2321)
changes later?

> Design a proper progress reporting & event listener API
> -------------------------------------------------------
>                 Key: SPARK-2321
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Java API, Spark Core
>    Affects Versions: 1.0.0
>            Reporter: Reynold Xin
>            Assignee: Josh Rosen
>            Priority: Critical
> This is a ticket to track progress on redesigning the SparkListener and JobProgressListener
> There are multiple problems with the current design, including:
> 0. I'm not sure if the API is usable in Java (there are at least some enums we used in
Scala and a bunch of case classes that might complicate things).
> 1. The whole API is marked as DeveloperApi, because we haven't paid a lot of attention
to it yet. Something as important as progress reporting deserves a more stable API.
> 2. There is no easy way to connect jobs with stages. Similarly, there is no easy way
to connect job groups with jobs / stages.
> 3. JobProgressListener itself has no encapsulation at all. States can be arbitrarily
mutated by external programs. Variable names are sort of randomly decided and inconsistent.

> We should just revisit these and propose a new, concrete design. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message