spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-8667) Improve Spark UI behavior at scale
Date Sat, 27 Jun 2015 06:01:05 GMT

     [ https://issues.apache.org/jira/browse/SPARK-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Wendell resolved SPARK-8667.
------------------------------------
    Resolution: Duplicate

> Improve Spark UI behavior at scale
> ----------------------------------
>
>                 Key: SPARK-8667
>                 URL: https://issues.apache.org/jira/browse/SPARK-8667
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>            Reporter: Patrick Wendell
>            Assignee: Shixiong Zhu
>
> This is a parent ticket and we can create child tickets when solving specific issues.
The main problem I would like to solve is the fact that the Spark UI has issues at very large
scale.
> The worst issue is when there is a stage page with more than a few thousand tasks. In
this case:
> 1. The page itself is very slow to load and becomes unresponsive with huge number of
tasks.
> 2. The Scala XML output can become so large that it crashes the driver program due to
OOM for a page with a huge number of tasks.
> I am not sure if (1) is caused by javascript slowness, or maybe just the raw amount of
data sent over the wire. If it is the latter, it might be possible to add compression to the
HTTP payload to help improve load time.
> It would be nice to reproduce+investigate these issues further and create specific sub
tasks to improve them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message