mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Mahler (JIRA)" <>
Subject [jira] [Commented] (MESOS-2735) Change the interaction between the slave and the resource estimator from polling to pushing
Date Mon, 01 Jun 2015 19:37:17 GMT


Benjamin Mahler commented on MESOS-2735:

Finally, one of the advantages of the pull model is that it's easier to reason about because
we don't have "anonymous" lambdas that cause execution in some other random place in the code
(i.e., you can easily see in the slave where the future that gets returned from `ResourceEstimator::estimate()`
gets handled). In addition, the ResourceEstimator remains "functional" in the sense that it
just has to return some value (or a future) from it's functions versus invoking some callback
that causes something to get run some other place (and in fact, may also block, so isn't it
safer for the ResourceEstimator to invoke the callback in it's own `async`?).

The invocation of the `ResourceEstimator::estimate()` followed by the `.then` is a nice pattern
that let's us compose with other things as well, which is harder to do with the lambda style
callbacks and why we've avoided it where we've been able (in fact, I'm curious which place
in the code are you imitating here?).

The anonymous lambdas are because we do not have an abstraction to represent an asychronous
stream of results:

// Returns a stream of the unused resources that can be oversubscribed.
Stream<Resources> unused();

This allows the caller to hold the stream and do composition on the next() Future result,
so we achieve push but we keep the ability to do composition. {{http::Pipe}} is a string-specialized
form of this and {{process::Queue<T>}} is an infinite-only version of {{process::Stream<T>}}.
An important additional property of {{process::Stream<T>}} is that it should provide
"asynchronous back-pressure" (e.g. unix pipes provide synchronous back-pressure by blocking
writes, however we need an asynchronous mechanism to back-pressure the writes), to control
the DoS concerns you guys have mentioned.

Take the allocator for example, it seems non-intuitive if the master had to continually call
a method to obtain the next allocation from {{Future<Allocation> allocation()}}. At
least, it requires more understanding than what it expressed in the return type. Whereas,
if we have a {{Stream<Allocation> initialize()}} or {{Stream<Allocation> allocations()}},
we're clearly capturing the semantics within the return type. Back-pressure is the key to
ensure that the master's speed of consumption limits the rate at which the allocator makes
allocations. Today, {{process::Queue<T>}} is an equivalent replacement since allocations
occur as an infinite stream and there is no back-pressuring.

> Change the interaction between the slave and the resource estimator from polling to pushing

> --------------------------------------------------------------------------------------------
>                 Key: MESOS-2735
>                 URL:
>             Project: Mesos
>          Issue Type: Bug
>            Reporter: Jie Yu
>            Assignee: Jie Yu
>              Labels: twitter
> This will make the semantics more clear. The resource estimator can control the speed
of sending resources estimation to the slave.
> To avoid cyclic dependency, slave will register a callback with the resource estimator
and the resource estimator will simply invoke that callback when there's a new estimation
ready. The callback will be a defer to the slave's main event queue.

This message was sent by Atlassian JIRA

View raw message