spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (SPARK-20656) Incremental parsing of event logs in SHS
Date Mon, 08 May 2017 18:22:04 GMT


Marcelo Vanzin commented on SPARK-20656:

This could also be considered related to or, if you squint a bit, a dupe of SPARK-6951.

> Incremental parsing of event logs in SHS
> ----------------------------------------
>                 Key: SPARK-20656
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.3.0
>            Reporter: Marcelo Vanzin
> This feature is mentioned in the spec attached to SPARK-18085 but there's not a lot of
discussion about it.
> It would be good to implement incremental parsing of event logs in the SHS. With the
new work, UI data is stored on disk, so it should be possible to save enough metadata about
the event log and the state of the listeners to allow one to resume parsing the log of a live
application at the point where it stopped in the previous iteration. 
> This would considerably speed up parsing on updates, and could be done speculatively
so that UIs for new applications are available in the SHS almost immediately.
> I'm filing this as a separate enhancement because I don't want to block SPARK-18085 on

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message