mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hans van den Bogert <hansbog...@gmail.com>
Subject Role-weights only honored with low allocation_interval
Date Tue, 26 Jan 2016 13:43:38 GMT
Hi, 

While investigating fairness possibilities with Mesos for Spark workloads I’m trying to
achieve for example a 4:1 weight ratio for two frameworks.
Imagine a system with two Spark frameworks (in fine-grained mode if you’re familiar with
Spark) and I want one the two frameworks to get four times more resources than the other when
both are contending for resources.

In mesos I set two roles “F1” “F2", with a weight of 4 and 1 respectively.

However during the times when both frameworks are in need of resources the latter gets close
to zero offers. Having read and more carefully investigated DRF I understood that memory is
the dominant resource in the case of framework 2 (F2) which Spark sets statically, i.e., it
doesn’t release once acquired, and in my case that is ~25% per slave.  So the allocator
thinks that F2 has received enough resources, since its dominant resource is already above
if weighted fair share. Thus all CPU offers go to framework 1 (F1).
To remedy this first hurdle I recalculate, although somewhat contrived, the ratio to 3.2 (
=80% / 25% ). 
After using the 3.2:1 ratio things are a bit better but still framework 2 (F2), during high
resource demand of both frameworks, only gets half of the resources it should get. 

At this point I was quite lost and tried changing several parameters, on of them was the allocation
interval (master option --allocation_interval) and set it to a relatively low 50ms instead
of the default 1000ms.
Suddenly my ratio was being honored perfectly and I was getting roughly a 4:1 CPU ratio between
the two Spark frameworks. (Verifying that my ratio 3.2:1, to circumvent spark’s static memory
allocation, was working. )

Perhaps it’s because I’m using only 10 physical nodes, however I made unit-tests in the
mesos-source to mimic my case, and there I could verify that the offers are made fairly according
to the weights.

Why is the fairness, expressed as being close as close to the defined role-weights, only honored
when the allocation interval is relatively low? Hope someone can explain the phenomenon. 


Thanks,

Hans
Mime
View raw message