mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jie Yu (JIRA)" <>
Subject [jira] [Commented] (MESOS-2818) Pass 'allocated' resources for each executor to the resource estimator.
Date Mon, 08 Jun 2015 19:06:02 GMT


Jie Yu commented on MESOS-2818:

Here is a new proposal. I personally think that's better than all existing proposals. Here
is the summary.

1) We kill the ResourceMonitor and move its logic (currently, only 'usages()') into the slave
Future<ResourceUsage> Slave::usage() { ... }

message ResourceUsage {
  message Executor {
    optional ExecutorInfo executor_info = 1;
    repeated Resource allocated = 2;
    repeated ResourceStatistics statistics = 3;

  repeated Resource total = 1; // Slave's total resources.
  repeated Executor executors = 2; // Per-executor allocated/usage information.

2) Slave collect per-executor allocated information when the resource estimator calls 'usage()':
Future<ResourceUsage> Slave::usage()
  ResourceUsage usage;
  list<Future<ResourceStatistics> futures;
  foreachvalue (const Framework* framework, frameworks) {
    foreachvalue (const Executor* executor, framework->executors) {
      ResourceUsage::Executor* entry = usage.add_executors()'

  return await(futures)
    .then(defer(self(), &Self::_usage, usage, lambda::_1));

3) In this way, it's guaranteed that if an executor has 'statistics', it will have 'allocated'
info available. Vice not versa. But in that case, we can just make conservative estimation
(i.e., assuming all resources allocated are consumed).

> Pass 'allocated' resources for each executor to the resource estimator.
> -----------------------------------------------------------------------
>                 Key: MESOS-2818
>                 URL:
>             Project: Mesos
>          Issue Type: Task
>            Reporter: Jie Yu
>            Assignee: Jie Yu
> Resource estimator obviously need this information to calculate, say the usage slack.
Now the question is how. There are two approaches:
> 1) Pass in the allocated resources for each executor through the 'oversubscribable()'
> 2) Let containerizer return total resources allocated for each container when 'usages()'
are invoked.
> I would suggest to take route (1) for several reasons:
> 1) Eventually, we'll need to pass in slave's total resources to the resource estimator
(so that RE can calculate allocation slack). There is no way that we can get that from containerizer.
The slave's total resources keep changing due to dynamic reservation. So we cannot pass in
the slave total resources during initialization.
> 2) The current implementation of usages() might skip some containers if it fails to get
statistics for that container (not an error). This will cause in-complete information to the
> 3) We may want to calculate 'unallocated = total - allocated' so that we can send allocation
slack as well. Getting 'total' and 'allocated' from two different components might result
in inconsistent value. Remember that 'total' keeps changing due to dynamic reservation.

This message was sent by Atlassian JIRA

View raw message