mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dario Rexin <dre...@apple.com>
Subject Re: MESOS-4694
Date Thu, 07 Jul 2016 15:40:01 GMT
Hi Joris,

I still don't really understand why we would parameterize SUPPRESS, to me that sounds like
a case for filters. The idea of SUPPRESS was to completely stop getting offers. 

Could you please explain why you think the patch is a hack? To me it just seems logical to
not sort frameworks that don't need to be considered in the allocator.

Thanks,
Dario

> On 07.07.2016, at 7:38 AM, Joris Van Remoortere <joris@mesosphere.io> wrote:
> 
> The reason that SUPPRESS doesn't just deactivate is because the intent was
> to be able to parameterize this call. At that point the change wouldn't
> work without turning this in to 2 cases.
> 
> I have asked to look at what a parameterized suppress would like and
> understand the performance impact of that before we do this.
> Have we reached consensus that there's no way to implement a generic
> parameterized suppress that is performant?
> 
> There are some refactorings that we had discussed with James, Jacob, and
> Ian that seem like lower hanging fruit. After those are made it might be
> worth reconsidering whether we need to do this hack.
> 
> 
> 
> —
> *Joris Van Remoortere*
> Mesosphere
> 
>> On Thu, Jul 7, 2016 at 10:15 AM, Guangya Liu <gyliu513@gmail.com> wrote:
>> 
>> Hi Ben and Dario,
>> 
>> The reason that we have "SUPPRESS" call is as following:
>> 1) Act as the complement to the current REVIVE call.
>> 2) The HTTP API do not have an API to "Deactivate" a framework, we want to
>> use "SUPPRESS", "DECLINE" and "DECLINE_INVERSE_OFFERS" to implement the
>> call for "DeactivateFrameworkMessage".
>> 
>> You can also refer to https://issues.apache.org/jira/browse/MESOS-3037 for
>> detail.
>> 
>> So I think that Dario's patch is good, we should remove the framework
>> clients when "SUPPRESS" and add the framework client back when "REVIVE". to
>> ignore those frameworks from sorter.
>> 
>> @Viond, any comments for this?
>> 
>> @Ben, for your concern of the benchmark test result is not easy to
>> understand, I have filed a JIRA ticket here
>> https://issues.apache.org/jira/browse/MESOS-5800 to trace.
>> 
>> Thanks,
>> 
>> Guangya
>> 
>> 
>> 
>>> On Thu, Jul 7, 2016 at 6:01 AM, Dario Rexin <drexin@apple.com> wrote:
>>> 
>>> Hi Vinod,
>>> 
>>> thanks for your reply. The reason it’s so much faster is because the
>>> sorting is a lot faster with fewer frameworks. Looping shouldn’t make a
>>> huge difference, as it used to just skip over the deactivated frameworks.
>>> 
>>> I don’t know what effects deactivating the framework in the master would
>>> have. The framework is still active and listening for events / sending
>>> calls. Could you please elaborate?
>>> 
>>> Thanks,
>>> --
>>>  Dario
>>> 
>>> On Jul 6, 2016, at 2:56 PM, Benjamin Mahler <bmahler@apache.org> wrote:
>>> 
>>> +implementer and shepherd of SUPPRESS
>>> 
>>> Is there any reason we didn't already just "deactivate" frameworks that
>>> were suppressing offers? That seems to be the natural implementation,
>>> performance aside, because the meaning of "deactivated" is: not being
>> sent
>>> any offers. The patch you posted seems to only take this half-way:
>> suppress
>>> = deactivation in the allocator, but not in the master.
>>> 
>>> Also, Dario it's a bit hard to interpret these numbers without reading
>> the
>>> benchmark code. My interpretation of these numbers is that this change
>>> makes the allocation loop complete more quickly when there are many
>>> frameworks that are in the suppressed state, because we have to loop over
>>> fewer clients. Is this an accurate interpretation?
>>> 
>>> On Wed, Jul 6, 2016 at 2:08 PM, Dario Rexin <drexin@apple.com> wrote:
>>> 
>>> Hi all,
>>> 
>>> I would like to revive https://issues.apache.org/jira/browse/MESOS-4694
>> <
>>> https://issues.apache.org/jira/browse/MESOS-4694>, especially
>>> https://reviews.apache.org/r/43666/ <https://reviews.apache.org/r/43666/
>>> .
>>> We heavily depend on this patch and would love to see it merged. To show
>>> the value of this patch, I ran the benchmark from
>>> https://reviews.apache.org/r/49616/ <https://reviews.apache.org/r/49616/
>>> 
>>> first on HEAD and then with the aforementioned patch applied. I took some
>>> lines out to make it easier to see the changes over time in the patched
>>> version and to keep this email shorter ;). I would love to get some
>>> feedback and discuss any necessary changes to get this patch merged.
>>> 
>>> Here are the results:
>>> 
>>> Mesos HEAD:
>>> 
>>> Using 2000 agents and 200 frameworks
>>> round 0 allocate took 3.064665secs to make 199 offers
>>> round 1 allocate took 3.029418secs to make 198 offers
>>> round 2 allocate took 3.091427secs to make 197 offers
>>> round 3 allocate took 2.955457secs to make 196 offers
>>> round 4 allocate took 3.133789secs to make 195 offers
>>> [...]
>>> round 50 allocate took 3.109859secs to make 149 offers
>>> round 51 allocate took 3.062746secs to make 148 offers
>>> round 52 allocate took 3.146043secs to make 147 offers
>>> round 53 allocate took 3.042948secs to make 146 offers
>>> round 54 allocate took 3.097835secs to make 145 offers
>>> [...]
>>> round 100 allocate took 3.027475secs to make 99 offers
>>> round 101 allocate took 3.021641secs to make 98 offers
>>> round 102 allocate took 2.9853secs to make 97 offers
>>> round 103 allocate took 3.145925secs to make 96 offers
>>> round 104 allocate took 2.99094secs to make 95 offers
>>> [...]
>>> round 150 allocate took 3.080406secs to make 49 offers
>>> round 151 allocate took 3.109412secs to make 48 offers
>>> round 152 allocate took 2.992129secs to make 47 offers
>>> round 153 allocate took 3.405642secs to make 46 offers
>>> round 154 allocate took 4.153354secs to make 45 offers
>>> [...]
>>> round 195 allocate took 3.10015secs to make 4 offers
>>> round 196 allocate took 3.029347secs to make 3 offers
>>> round 197 allocate took 2.982825secs to make 2 offers
>>> round 198 allocate took 2.934595secs to make 1 offers
>>> round 199 allocate took 313212us to make 0 offers
>>> 
>>> Mesos HEAD + allocator patch:
>>> 
>>> Using 2000 agents and 200 frameworks
>>> round 0 allocate took 3.248205secs to make 199 offers
>>> round 1 allocate took 3.170852secs to make 198 offers
>>> round 2 allocate took 3.135146secs to make 197 offers
>>> round 3 allocate took 3.143857secs to make 196 offers
>>> round 4 allocate took 3.127641secs to make 195 offers
>>> [...]
>>> round 50 allocate took 2.492077secs to make 149 offers
>>> round 51 allocate took 2.435054secs to make 148 offers
>>> round 52 allocate took 2.472204secs to make 147 offers
>>> round 53 allocate took 2.457228secs to make 146 offers
>>> round 54 allocate took 2.413916secs to make 145 offers
>>> [...]
>>> round 100 allocate took 1.645015secs to make 99 offers
>>> round 101 allocate took 1.647373secs to make 98 offers
>>> round 102 allocate took 1.619147secs to make 97 offers
>>> round 103 allocate took 1.625496secs to make 96 offers
>>> round 104 allocate took 1.580513secs to make 95 offers
>>> [...]
>>> round 150 allocate took 1.064716secs to make 49 offers
>>> round 151 allocate took 1.065604secs to make 48 offers
>>> round 152 allocate took 1.053049secs to make 47 offers
>>> round 153 allocate took 1.041333secs to make 46 offers
>>> round 154 allocate took 1.0461secs to make 45 offers
>>> [...]
>>> round 195 allocate took 569640us to make 4 offers
>>> round 196 allocate took 562107us to make 3 offers
>>> round 197 allocate took 547632us to make 2 offers
>>> round 198 allocate took 530765us to make 1 offers
>>> round 199 allocate took 24426us to make 0 offers
>>> 
>>> --
>>>  Dario
>> 

Mime
View raw message