Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 15 Nov 2016 01:09:58 +0000 (UTC)
From: "Marcelo Vanzin (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13014927.1477351921000.275734.1479172198530@Atlassian.JIRA>
In-Reply-To: <JIRA.13014927.1477351921000@Atlassian.JIRA>
References: <JIRA.13014927.1477351921000@Atlassian.JIRA> <JIRA.13014927.1477351921876@arcas>
Subject: [jira] [Commented] (SPARK-18085) Better History Server scalability
 for many / large applications
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 15 Nov 2016 01:10:05 -0000


    [ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665618#comment-15665618 ] 

Marcelo Vanzin commented on SPARK-18085:
----------------------------------------

Hey all, any more feedback here?

I've made some progress in the M3 implementation, but I've been distracted by more urgent issues, and it's also a pretty large milestone. But it still would be good to know whether people have any concerns about the changes being proposed before I spent too much time working on something that people just don't want.

Just be aware that where the data is written to is not really the most important things here; the main thing is that UI data would now live somewhere that is not the Spark driver's memory, and most of the changes are towards implementing the abstractions to support that. In fact, you could even still store things in memory later on too, although I don't think that will be necessary (nor desired).

> Better History Server scalability for many / large applications
> ---------------------------------------------------------------
>
>                 Key: SPARK-18085
>                 URL: https://issues.apache.org/jira/browse/SPARK-18085
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Spark Core, Web UI
>    Affects Versions: 2.0.0
>            Reporter: Marcelo Vanzin
>         Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues. I'll be attaching a document shortly describing the issues and suggesting a path to how to solve them.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org