mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Gaudio <adgau...@gmail.com>
Subject Re: Setting minimum offer size
Date Wed, 24 Jun 2015 15:31:08 GMT
We have struggled with this problem many times, and our solution was to
reduce the size of jobs.  Reducing job size has many advantages:

- Big jobs have less chance of getting starved out
- Frameworks have less incentive to hold onto offers
- The DRF algorithm assumes that all frameworks want offers equally (unless
you use weighting).  This means that if a framework with large jobs
receives a lot of small offers it can't use, and it is therefore not only
starved itself, but also starving out the smaller jobs.  This leads to more
under-utilization of the cluster.


There are other ways to solve this problem, besides reducing job size.
These are probably more complicated, and I'd love if others have good
solutions or comments here:

  - Statically partition the cluster such that some machines serve large
jobs (and perhaps some small jobs with uniqueness constraints that
guarantee small jobs won't all cluster on one host).
  - Write your own variant of the Allocator module (not just the Sorter
module) that gives preference to large offers.  I think you might have to
modify the sort() api call
<https://github.com/apache/mesos/blob/2ef849a7ca91a2785d950141a6bc1819d40a75d3/src/master/allocator/mesos/hierarchical.hpp#L897>
to include the size of an offer.  This generally seems hard.
  - Wait for optimistic offers or some other planned Mesos feature and hope
the community will somehow figure this out?


Does anyone have other ideas?  As I mentioned above, my solution was to
make all frameworks generate small jobs, where small is relative to the
amount of resources on a given slave.

Thanks,
Alex Gaudio



On Wed, Jun 24, 2015 at 10:44 AM Brian Candler <b.candler@pobox.com> wrote:

> On 24/06/2015 15:35, David Greenberg wrote:
> > I'm not aware of any minimum offer size option in mesos. What I've
> > seen success with is holding onto small offers and waiting until I
> > accumulate enough to launch the large task. This way, the need for
> > large offers doesn't affect the cluster, but the framework that needs
> > it gets it.
> This means that the execution of a task doesn't need to be linked to a
> single resource offer, but they can be aggregated?
>
> Also, it seems to me this would break down if there were two frameworks
> both which required 20GB of RAM. Let's say each of them grabs 16 x 1GB
> offers. At this point all 32GB of one node has been committed, and there
> is deadlock.
>
> Conversely, ISTM that a 20GB minimum offer size would lead to big
> wastage in the case where frameworks only made 1GB allocations. After
> 13GB of RAM had been used (19GB free), no further offers would be made.
>
> Regards,
>
> Brian.
>
>

Mime
View raw message