mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Felix Bechstein (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-2353) Improve performance of the master's state.json endpoint for large clusters.
Date Tue, 10 Nov 2015 14:14:11 GMT

    [ https://issues.apache.org/jira/browse/MESOS-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14998634#comment-14998634
] 

Felix Bechstein commented on MESOS-2353:
----------------------------------------

We patched the master to hold only 100 completed tasks per framework and 10 completed frameworks.
It reduced the state size to ~2MB but the master is still using all its CPU to generate.

We stopped fetching the /master/state from all the browsers of your developers with iptables
and the load was gone.

> Improve performance of the master's state.json endpoint for large clusters.
> ---------------------------------------------------------------------------
>
>                 Key: MESOS-2353
>                 URL: https://issues.apache.org/jira/browse/MESOS-2353
>             Project: Mesos
>          Issue Type: Improvement
>          Components: master
>            Reporter: Benjamin Mahler
>              Labels: newbie, scalability, twitter
>
> The master's state.json endpoint consistently takes a long time to compute the JSON result,
for large clusters:
> {noformat}
> $ time curl -s -o /dev/null localhost:5050/master/state.json
> Mon Jan 26 22:38:50 UTC 2015
> real	0m13.174s
> user	0m0.003s
> sys	0m0.022s
> {noformat}
> This can cause the master to get backlogged if there are many state.json requests in
flight.
> Looking at {{perf}} data, it seems most of the time is spent doing memory allocation
/ de-allocation. This ticket will try to capture any low hanging fruit to speed this up. Possibly
we can leverage moves if they are not already being used by the compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message