spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhoukang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-20582) Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing
Date Wed, 03 May 2017 16:06:05 GMT
zhoukang created SPARK-20582:
--------------------------------

             Summary: Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing
                 Key: SPARK-20582
                 URL: https://issues.apache.org/jira/browse/SPARK-20582
             Project: Spark
          Issue Type: Improvement
          Components: Deploy
    Affects Versions: 2.1.0
            Reporter: zhoukang
            Priority: Critical


In the current code of HistoryServer,jetty server will be started after all the logs fetch
from yarn has been replayed.However,when the number of logs is becoming larger,the start time
of jetty will be too long.
Here,we implement a solution which using ApplicationAttemptInfo checkpointing to speed up
the start of historyserver.When historyserver is starting,it will load ApplicationAttemptInfo
from checkpoint file first if exists which is faster then replaying one by one.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message