mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Povilas Versockas <p.versoc...@gmail.com>
Subject Re: Question on dynamic reservations
Date Wed, 18 Jan 2017 09:58:53 GMT
Thanks for information and ideas! I think labeled static reservation would
help if framework could modify labels during runtime.

Personally I think statically reserved resources should be offered only to
frameworks but look like free (role:*) resources. Then framework developers
could use labeled dynamic reservations and there won’t be any code
differences handling static reservations and dynamic reservations. And this
would make current frameworks like Cassandra or dcos-commons library
support static reservation.

Given the current situation it looks like quota may be a solution for me.
The current idea is to create a custom resource on my mesos-agents and set
a quota.

Example:

Set mesos-agent with flag:

--resources
cpus(*):8;mem(*):4096;disk(*):4096;ports(*):[31000-32000];mysql(*):1


And then using operators API set quota:

{

   "role": “role”,

   "guarantee": [

     {

       "name": "cpus",

       "type": "SCALAR",

       "scalar": { "value": 8 }

     },

     {

       "name": "mem",

       "type": "SCALAR",

       "scalar": { "value": 4096 }

     },

…

     {

       "name": “mysql”,

       "type": "SCALAR",

       "scalar": { "value": 1 }

     }

   ]

 }


This should make mesos-master save agent’s resources for my role and let me
use dynamic reservations.



On Wed, Jan 18, 2017 at 2:29 AM, Greg Mann <greg@mesosphere.io> wrote:

> Thanks Gabriel, that makes sense. It sounds like labels on static
> reservations might be the most expedient path toward a solution to this
> problem, but that is not without its complications, as suggested in the
> related ticket which Neil filed a while back: https://issues.apache.
> org/jira/browse/MESOS-4476
>
> Povilas, also see this related ticket that Gabriel pointed me to:
> https://issues.apache.org/jira/browse/MESOS-6939
>
> It sounds like this is a real issue for stateful framework developers, so
> hopefully we will find some time soon to implement a solution. In the
> meantime, Povilas, I'm afraid to say I don't know exactly what solution to
> recommend. If anybody else in the community has some ideas, it would be
> great to hear them :)
>
> Cheers,
> Greg
>
>
> On Tue, Jan 17, 2017 at 2:52 PM, Gabriel Hartmann <gabriel@mesosphere.io>
> wrote:
>
>> @Greg: The reason people use static reservation is to enforce that
>> particular resources (usually disks) can only be consumed by a particular
>> framework.  They also don't know when the stateful service is going to be
>> installed necessarily so they don't want to race with other frameworks to
>> consume those special resources.  So static reservation is desirable.
>> However, all stateful services also need more information about reserved
>> resources than is natively provided by Mesos in the static reservation case
>> (i.e. the labels he describes).  `dcos-commons` does the same thing.
>> Various work arounds exist, but none are able to provide resource
>> allocation enforcement because only roles do that.  An alternate resource
>> allocation enforcement mechanism is needed.  Usually this is the part where
>> people start talking about quota.
>>
>> Neither option 1 nor option 2 provided a race proof way to get fully
>> labeled reserved resources.  It's been proposed in the past that it be
>> allowed to add labels to statically reserved resources.  That's kind of
>> fine except now you have these things that can't really be UNRESERVEd but
>> look exactly like dynamic resources which can...
>>
>> Quota w/ chunks as a step in the deployment of stateful services is very
>> desirable in an adversarial environment.  However if your'e in a
>> cooperative environment (i.e. you're not in an adversarial relationship
>> with other frameworks) if you had resources (particularly disk resources)
>> with attributes on them you could have frameworks voluntarily choose not to
>> consume resources not meant for them.
>>
>> e.g. Disk resource has attribute `CASSANDRA`.  Ok, since I'm a Kafka
>> framework I won't go use that disk.
>>
>> On Tue, Jan 17, 2017 at 11:24 AM Greg Mann <greg@mesosphere.io> wrote:
>>
>>> Hi Povilas,
>>> Another approach you could try is to use dynamic reservations only. You
>>> could either:
>>>
>>>    1. Alter your stateful framework to dynamically reserve the
>>>    resources that it needs, or
>>>    2. Add a script to your cluster tooling that would make use of the
>>>    operator endpoint for dynamic reservations [1]
>>>    <http://mesos.apache.org/documentation/latest/reservation/> to
>>>    dynamically reserve the stateful framework's resources when your cluster is
>>>    initially provisioned. This would have a similar effect to static
>>>    reservations, but would allow you to set labels
>>>
>>> Approach #1 makes sense to me; is there a reason that it's not feasible
>>> for your stateful framework to dynamically reserve its own resources? This
>>> is the typical workflow that I would recommend. I'm not too familiar with
>>> Aurora, so perhaps it's adding some complexity that I'm unaware of?
>>>
>>> Cheers,
>>> Greg
>>>
>>> [1] http://mesos.apache.org/documentation/latest/reservation/
>>>
>>>
>>> On Tue, Jan 17, 2017 at 12:28 AM, Povilas Versockas <
>>> p.versockas@gmail.com> wrote:
>>>
>>> Hey,
>>>
>>> Thanks for writing me back!
>>>
>>> Maybe there is some other method to solve this problem on statically
>>> reserved cluster? The solution could be making agent's resources appear as
>>> unreserved resources to only selected framework. I can see that mesos-agent
>>> has --acls flag, so maybe tinkering with this could help me. Of course it
>>> is possible to implement this in the framework scheduler, but this will add
>>> way more clunkiness to the code. It feels like this kind of resource
>>> management should be part of Mesos. Maybe I'm missing something?
>>>
>>>
>>>
>>> On Mon, Jan 16, 2017 at 4:58 PM, haosdent <haosdent@gmail.com> wrote:
>>>
>>> Hi, @Povilas It is possible to dynamic reserve unreserved resources on
>>> those agents.
>>>
>>> On Fri, Jan 13, 2017 at 2:47 PM, Povilas Versockas <
>>> p.versockas@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> Maybe someone can help me with a problem I'm having. Short version of
>>> the question is:
>>> Is it possible to use dynamic reservation on statically reserved Mesos
>>> agents?
>>>
>>> The current situation is that we have Mesos cluster which runs many
>>> frameworks (aurora, spark, cassandra) and we are developing a custom
>>> framework for stateful tasks. Our framework manages stateful tasks for many
>>> users. Currently we statically reserved our hardware which has good disks
>>> only to be used by our framework (via --resources flag on Mesos Agents).
>>>
>>> The problem we are facing is that if one stateful task fails we would
>>> like to relaunch it on the same host with the same port, cpu, disk and
>>> memory.
>>> With dynamic reservations we would put a label with task id on a
>>> reservation and on failure would just simply reuse the reserved offer.
>>> On the other hand with statically reserved Mesos agents we cannot put
>>> any labels and so we cannot distinguish offers which should have been
>>> reserved for a task and a new offer.
>>> This leaves us in the situation that if one stateful task fails and
>>> there are new stateful tasks, the new tasks can be scheduled on failed
>>> task's Mesos agent, filling it up and taking it's port, cpu and memory.
>>>
>>>
>>> --
>>> Regards
>>> Povilas Versockas
>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>>
>>>
>>>
>>> --
>>> Pagarbiai
>>> Povilas Versockas
>>>
>>>
>>>
>


-- 
Pagarbiai
Povilas Versockas

Mime
View raw message