mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Marshall" <twm...@gmail.com>
Subject Re: Review Request: Made DRF per user and log(N)
Date Thu, 12 Jul 2012 23:11:19 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5599/
-----------------------------------------------------------

(Updated July 12, 2012, 11:11 p.m.)


Review request for mesos and Benjamin Hindman.


Changes
-------

Updated to trunk.


Description
-------

Rewrote the DRF algorithm in DominantShareAllocator to calculate shares on a per user, rather
than per framework, basis and to store those shares in sorted order so that allocations have
log(n) time complexity instead of n^2.

Significant restructuring of the allocator code to make writing new allocators easier:

- The Allocator class is renamed AllocatorProcess.
- A new Allocator class is introduced which knows only about "elements" (either user names
or frameworkIds) and how many resources they have been allocated, and then has an interface
for retrieving the elements in the order they should be allocated to.
- SimpleAllocatorProcess which can be used with different Allocators to carry out different
allocation policies, including the possibility of having different policies for the per-user
and per-framework allocations.

Why is this restructuring a good thing?
Now that we're doing allocations based on users, rather than frameworks, we have to do two
levels of allocation - determine which user to give resources to, then determine which of
their frameworks to actually make the offer to. Rather than hard-coding the allocation policies
in, the new Allocator class allows us to reuse the same code to do both levels of allocation,
and it allows us to easily swap different policies in or mix and match. You can even imagine
some day having an allocator that allows each user to pick the per-framework allocation policy
they prefer.

Merely giving SimpleAllocatorProcess different Allocators is not sufficient to implement all
allocation policies - only those where all resources are considered equal and there is no
revocation. But, you can always create new AllocatorProcess subclasses (there will soon be
a code review where I implement a static allocator this way) and these new AllocatorProcess
classes can still reuse Allocators as appropriate.

It also makes testing the allocator logic a lot easier since we can instantiate Allocators,
give them arbitrary elements, and check that they return the right thing, instead of worrying
about instantiating slaves/frameworks and using mock objects.


This addresses bugs MESOS-225 and MESOS-226.
    https://issues.apache.org/jira/browse/MESOS-225
    https://issues.apache.org/jira/browse/MESOS-226


Diffs (updated)
-----

  src/Makefile.am 46285ac 
  src/local/local.hpp 8ae5b9e 
  src/local/local.cpp 46537d7 
  src/master/allocator.hpp 38736cb 
  src/master/allocator.cpp PRE-CREATION 
  src/master/allocator_process.hpp PRE-CREATION 
  src/master/dominant_share_allocator.hpp 6587f33 
  src/master/dominant_share_allocator.cpp 1ea29c7 
  src/master/main.cpp 8d38fb1 
  src/master/master.hpp 6cb4810 
  src/master/master.cpp edefd70 
  src/master/simple_allocator_process.hpp PRE-CREATION 
  src/master/simple_allocator_process.cpp PRE-CREATION 
  src/tests/allocator_tests.cpp 610826b 
  src/tests/fault_tolerance_tests.cpp dd578aa 
  src/tests/master_detector_tests.cpp 758c8b9 
  src/tests/master_tests.cpp b586984 
  src/tests/resource_offers_tests.cpp d06cae2 
  src/tests/slave_tests.cpp e9b25ba 
  src/tests/utils.hpp 1062dfc 

Diff: https://reviews.apache.org/r/5599/diff/


Testing
-------

make check on Lion


Thanks,

Thomas Marshall


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message