spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications
Date Tue, 15 Nov 2016 01:09:58 GMT


Marcelo Vanzin commented on SPARK-18085:

Hey all, any more feedback here?

I've made some progress in the M3 implementation, but I've been distracted by more urgent
issues, and it's also a pretty large milestone. But it still would be good to know whether
people have any concerns about the changes being proposed before I spent too much time working
on something that people just don't want.

Just be aware that where the data is written to is not really the most important things here;
the main thing is that UI data would now live somewhere that is not the Spark driver's memory,
and most of the changes are towards implementing the abstractions to support that. In fact,
you could even still store things in memory later on too, although I don't think that will
be necessary (nor desired).

> Better History Server scalability for many / large applications
> ---------------------------------------------------------------
>                 Key: SPARK-18085
>                 URL:
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Spark Core, Web UI
>    Affects Versions: 2.0.0
>            Reporter: Marcelo Vanzin
>         Attachments: spark_hs_next_gen.pdf
> It's a known fact that the History Server currently has some annoying issues when serving
lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues. I'll be attaching
a document shortly describing the issues and suggesting a path to how to solve them.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message