mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neil Conway <neil.con...@gmail.com>
Subject Re: Review Request 57527: Avoided storing weights in the allocator.
Date Fri, 17 Mar 2017 19:48:11 GMT


> On March 17, 2017, 1:41 a.m., Benjamin Mahler wrote:
> > Thinking a little more about this, it seems odd that the sorter is storing weight
information for clients that are unknown to it. Why? It seems cleaner if the sorter only understands
weights for its clients, rather than potential future clients.
> > 
> > One way to resolve this would be to have roles with weights (but no allocations
or frameworks subscribed to them) be present as inactive clients in the sorters, and using
the existing interface. That seems to be in line with the approach in the next patch?
> 
> Neil Conway wrote:
>     _Something_ needs to store weights for potential future clients. Can you elaborate
why you think it is cleaner for this knowledge to live in the allocator versus in the sorter?
Weights are actually a property of the sorter; other than passing the appropriate weight configuration
to the sorters, the allocator doesn't actually care about weights.
>     
>     The next patch does not represent weights on internal nodes by using inactive clients.
Inactive clients still correspond to _sorter clients_; you can set a weight on an arbitrary
role path, which might not be associated with any client (see the `TODO` around renaming the
`Client` struct to reflect this). Once you introduce hierarchical roles, the set of names
which have a weight defined and the set of client names might have very little in common.
> 
> Benjamin Mahler wrote:
>     > Can you elaborate why you think it is cleaner for this knowledge to live in
the allocator versus in the sorter?
>     
>     I think the confusing aspect of this is that the sorter is "anticipating" future
clients that may never arrive and have no effect on the sorter unless at some point they "arrive",
which means we're storing unnecessary information as far as the sorter's responsibilities
are concerned. As another point of confusion here, it seems that we can only "anticipate"
some of the time:
>     
>     (1) If a role has a non-default weight, we can anticipate that it is a potential
client.
>     (2) If the role does not have a default weight, we can't anticipate it's potential
as a client (i.e. all valid, authorized roles are potential clients).
>     (3) We can't anticipate framework ids as clients.
>     
>     Something needs to store the configuration and I think it seems a mesos component
that understands roles is more suited to that.
>     
>     > Once you introduce hierarchical roles, the set of names which have a weight
defined and the set of client names might have very little in common.
>     
>     It sounds like there are three states: (`potential client`, `inactive client`, `active
client`)? Is it possible to avoid the potential client complication by treating potential
clients the same as inactive clients in the hierarchical sorter changes? Why do we need to
call out potential clients distinctly from inactive clients when the hierarchy is introduced?

> I think the confusing aspect of this is that the sorter is "anticipating" future clients
that may never arrive and have no effect on the sorter unless at some point they "arrive",
which means we're storing unnecessary information as far as the sorter's responsibilities
are concerned.

To me, weights are configuration settings for the sorter. A given configuration setting may
or may not influence the behavior of a component; personally, I don't find that to be confusing.
We're only storing "unnecessary information" if we can perfectly predict the future set of
sorter clients, which we cannot.

> Something needs to store the configuration and I think it seems a mesos component that
understands roles is more suited to that.

Can you elaborate on why you think it is more suitable for this config to live in a component
that understands roles?

> It sounds like there are three states: (potential client, inactive client, active client)

I'm not sure this really captures the situation once hierarchy is introduced: setting a weight
can influence allocation behavior, even if the "potential client" never materializes.


- Neil


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57527/#review169247
-----------------------------------------------------------


On March 17, 2017, 1:12 a.m., Neil Conway wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57527/
> -----------------------------------------------------------
> 
> (Updated March 17, 2017, 1:12 a.m.)
> 
> 
> Review request for mesos, Benjamin Bannier, Benjamin Mahler, and Michael Park.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Previously, DRFSorter only kept track of weights for clients currently
> in the sorter; the allocator supplied the current weight when adding a
> client to the sorter.
> 
> This commit changes the sorter to keep track of all weights, not just
> those for the active clients in the sorter. The allocator can now just
> pass along role weights to the role sorters, rather than needing to
> track them itself.
> 
> This commit changes the sorter API.
> 
> 
> Diffs
> -----
> 
>   src/master/allocator/mesos/hierarchical.hpp f84b0574ce9a392c9528c87b04b01dbb2053cff7

>   src/master/allocator/mesos/hierarchical.cpp 8d54a8cca1bb478f4437f68c5e14f66a9f9bb9e9

>   src/master/allocator/sorter/drf/sorter.hpp 76329220e1115c1de7810fb69b943c78c078be59

>   src/master/allocator/sorter/drf/sorter.cpp ed54680cecb637931fc344fbcf8fd3b14cc24295

>   src/master/allocator/sorter/sorter.hpp b3029fcf7342406955760da53f1ae736769f308c 
>   src/tests/sorter_tests.cpp ec0636beb936d46a253d19322f2157abe95156b6 
> 
> 
> Diff: https://reviews.apache.org/r/57527/diff/3/
> 
> 
> Testing
> -------
> 
> `make check`
> 
> 
> Thanks,
> 
> Neil Conway
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message