aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Farner <wfar...@apache.org>
Subject Re: Multiple executor support
Date Sun, 08 Nov 2015 18:50:05 GMT
>
> all standard mesos inputs are available to the custom executor which is
> driven from client
>

This is the part i'm not convinced is a good idea due to security issues.
The unix user of the executor (via setuid) is set at the framework level
[1].  This would force you to use a completely unprivileged user for the
whole cluster to avoid escalation issues.

Perhaps further down the line we could create an admin API for
adding/removing executors, however.

[1]
https://github.com/apache/mesos/blob/1e7be5ca417871828e21c38de07142ee9b2c772e/include/mesos/mesos.proto#L208-L212

On Sun, Nov 8, 2015 at 10:18 AM, <meghdoot_b@yahoo.com> wrote:

> I see. That would be excellent Bill if you are willing to take on this and
> actively drive it.
>
> As you mention if all standard mesos inputs are available to the custom
> executor which is driven from client that would be great. URI set for files
> in sandbox, data blob (could be anything that an executor wants from
> client), set of dynamic ports send as resources to executor from scheduler
> as indicated by client (some convention defined in client request that is
> understood by scheduler), env variables etc that I can think of.
>
> Thx
>
> Sent from my iPhone
>
> On Nov 7, 2015, at 7:39 PM, Bill Farner <wfarner@apache.org> wrote:
>
> Could you kindly elaborate with an example how the end user can pass in
>> the executor of their choice in the API . I remember that other than a name
>> passed on the API there was no scope to pass in the path of the custom
>> executor or things like Fetcher URI sets that can be downloaded by mesos in
>> the sandbox.
>
>
> The user submitting jobs cannot do this, but the cluster administrator can
> specify an arbitrary executor for the cluster (which is effectively done
> today by setting the URI of the thermos executor).  In other words, i can
> use a non-thermos executor by using the scheduler thrift API so long as the
> executor data field is compatible with the executor referenced by the
> scheduler command line.
>
> At any rate, i think adding support for custom executor fields (mirroring
> mesos protobuf fields) is valuable.  I would like to start there, and at
> that point adding support for multiple executors becomes trivial.
>
> I'm happy to take this work on myself, though it will probably be another
> week or two before patches start landing.  I already have a patch locally
> that does a bunch of necessary prep work, however.
>
> Thanks for pushing on this feature!
>
>
> On Wed, Nov 4, 2015 at 8:50 AM, <meghdoot_b@yahoo.com> wrote:
>
>> Could you kindly elaborate with an example how the end user can pass in
>> the executor of their choice in the API . I remember that other than a name
>> passed on the API there was no scope to pass in the path of the custom
>> executor or things like Fetcher URI sets that can be downloaded by mesos in
>> the sandbox.
>> I remember we discussed to make the change like marathon where one can
>> pass in all details of the executor from client but you had opposed citing
>> security. That should be captured in the ticket. Hence, the patch was made
>> to load up selected executors from the scheduler side and depending on the
>> executor name passed in the API the relevant executor configs can be used
>> and filled in task.
>>
>> So kind of confused. Examples will help.
>>
>> Thx
>>
>> Sent from my iPhone
>>
>> On Nov 4, 2015, at 8:17 AM, Bill Farner <wfarner@apache.org> wrote:
>>
>> My change in stance partially came from the realization that the
>> scheduler is not currently coupled to the executor implementation.  As a
>> result, an API consumer (bypassing the client) could already use any
>> executor implementation they choose.  This could be interpreted as custom
>> executor support, and i would like to know if that satisfies the general
>> use case.
>>
>> I am slightly uneasy about supporting user-selected executors on the
>> scheduler simply because of the complexity that comes with it (the
>> configuration complexity is one, but there will also likely be a need for
>> ACLs, and even greater complexity in the client to support
>> multiple/arbitrary configuration input formats).
>>
>> So, my question becomes - is this (end-user-selected executors) a real
>> use case today, or a hypothetical one?  If it a real use case, some
>> elaboration would be useful to make sure we are building the right software
>> for it.
>>
>> On Tue, Nov 3, 2015 at 9:30 PM, <meghdoot_b@yahoo.com> wrote:
>>
>>> Bill what do you think?
>>>
>>> Thx
>>>
>>> Sent from my iPhone
>>>
>>> > On Nov 2, 2015, at 1:00 PM, Zameer Manji <zmanji@apache.org> wrote:
>>> >
>>> > +wfarner
>>> >
>>> > I believe Bill was heavily involved in reviewing the proposed patch and
>>> > design. Bill, care to comment on what you think here?
>>> >
>>> >> On Mon, Nov 2, 2015 at 12:55 PM, <meghdoot_b@yahoo.com.invalid>
>>> wrote:
>>> >>
>>> >> Do we have a decision on this?
>>> >>
>>> >>
>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/AURORA-1376
>>> >>
>>> >> It would help to know where we stand on this.
>>> >>
>>> >> Thx
>>> >>
>>> >>
>>> >> Sent from my iPhone
>>> >>
>>> >> --
>>> >> Zameer Manji
>>> >>
>>> >>
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message