mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Mahler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-9234) Improve performance of offer fan-out in the master.
Date Sat, 15 Sep 2018 20:54:00 GMT

    [ https://issues.apache.org/jira/browse/MESOS-9234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616511#comment-16616511
] 

Benjamin Mahler commented on MESOS-9234:
----------------------------------------

Reduced excessive map lookups:
{noformat}
commit 7e7ef60d265382e64dab3f93e8d5f101ab223f21
Author: Benjamin Mahler <bmahler@apache.org>
Date:   Wed Sep 5 14:50:53 2018 -0700

    Avoided triple-lookup of the framework in Master::offer.

    Review: https://reviews.apache.org/r/68635
{noformat}

{noformat}
commit 804b4a59b17a0749f80cb2010630d7907a148683
Author: Benjamin Mahler <bmahler@apache.org>
Date:   Wed Sep 5 14:49:40 2018 -0700

    Avoided double-lookup in Master::getFramework.

    Review: https://reviews.apache.org/r/68634
{noformat}

Reserve vector capacity where possible:
{noformat}
commit 82dbbc9d90079fd5134a28377e21669b2f41980e
Author: Benjamin Mahler <bmahler@apache.org>
Date:   Wed Sep 5 14:51:39 2018 -0700

    Reserved vector capacities within Master::offer.

    Review: https://reviews.apache.org/r/68636
{noformat}

Reduced copying in offer handling:
{noformat}
commit 866cc56cea88b091068e98447fa50a6483283469
Author: Benjamin Mahler <bmahler@apache.org>
Date:   Wed Sep 5 14:52:45 2018 -0700

    Avoided extra copying of Resources in Master::offer.

    In order to remove ephemeral ports resources, we now perform an
    efficient swap to last + remove last operation.

    Review: https://reviews.apache.org/r/68637
{noformat}

Avoid double copying of master -> framework messages (see MESOS-9213):
{noformat}
commit 898349458e1a95c52a2f347ed997d381d36f22f3
Author: Benjamin Mahler <bmahler@apache.org>
Date:   Thu Sep 6 15:31:48 2018 -0700

    Avoided double copying of outgoing messages in the master.

    When framework metrics were introduced, there was an evolve + devolve
    conversion occuring in order to map old style messages to the new
    `scheduler::Event::Type`. This results in copying all outgoing
    messages twice.

    This patch avoids the copying by adding overloads for the old style
    messages.

    Review: https://reviews.apache.org/r/68662
{noformat}

Reduced copying when evolving the ResourceOffersMessage:
{noformat}
commit 787f6c64c06be685c2e073467c440cb5c5fc0d89
Author: Benjamin Mahler <bmahler@apache.org>
Date:   Tue Sep 11 17:53:38 2018 -0700

    Reduced protobuf copying in evolve helpers.

    This uses move assignment rather than CopyFrom, where applicable.

    Review: https://reviews.apache.org/r/68695
{noformat}

{noformat}
commit 34bc7d6ae18f8221361e77e2f557e2a5e07380b8
Author: Benjamin Mahler <bmahler@apache.org>
Date:   Fri Sep 14 18:34:05 2018 -0700

    Avoid an unnecessary copy of evolved repeated fields.

    The repeated field evolve overload unnecessarily copies the input
    repeated field.
{noformat}

{noformat}
commit 4805a47c689a6aa8bdc184e64517f77d210f5611 (HEAD -> master, apache/master)
Author: Benjamin Mahler <bmahler@apache.org>
Date:   Sat Sep 15 13:29:48 2018 -0700

    Reserved capacity when evolving repeated fields.
{noformat}

> Improve performance of offer fan-out in the master.
> ---------------------------------------------------
>
>                 Key: MESOS-9234
>                 URL: https://issues.apache.org/jira/browse/MESOS-9234
>             Project: Mesos
>          Issue Type: Improvement
>          Components: allocation, master
>            Reporter: Benjamin Mahler
>            Priority: Major
>
> Of interest to improving scalability is the ability for the master to quickly fan out
offers as the allocator makes new allocation decisions.
> Some low hanging fruit:
> * Low hanging copy elimination.
> * Send more compact and cache-friendlier data structures across master <-> allocator
interface boundary (right now the allocator makes a call back into the master with a double-nested
hashmap<role,hashmap<SlaveID,Resources>>).
> * Reduce excessive map lookups.
> Longer term:
> * Update v1 message evolution paths to avoid de-serializing / serializing in favor of
copying fields (either manually or with a reflection based copier), or by moving individual
fields when possible (this will only help with string fields AFAICT).
> * Update the master <-> allocator interface to send all framework allocations in
one call, and use a parallel loop (see MESOS-8587) in the master to speed up the fan-out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message