spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <>
Subject [jira] [Resolved] (SPARK-8667) Improve Spark UI behavior at scale
Date Sat, 27 Jun 2015 06:01:05 GMT


Patrick Wendell resolved SPARK-8667.
    Resolution: Duplicate

> Improve Spark UI behavior at scale
> ----------------------------------
>                 Key: SPARK-8667
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>            Reporter: Patrick Wendell
>            Assignee: Shixiong Zhu
> This is a parent ticket and we can create child tickets when solving specific issues.
The main problem I would like to solve is the fact that the Spark UI has issues at very large
> The worst issue is when there is a stage page with more than a few thousand tasks. In
this case:
> 1. The page itself is very slow to load and becomes unresponsive with huge number of
> 2. The Scala XML output can become so large that it crashes the driver program due to
OOM for a page with a huge number of tasks.
> I am not sure if (1) is caused by javascript slowness, or maybe just the raw amount of
data sent over the wire. If it is the latter, it might be possible to add compression to the
HTTP payload to help improve load time.
> It would be nice to reproduce+investigate these issues further and create specific sub
tasks to improve them.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message