aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From meghdoo...@yahoo.com.INVALID
Subject Re: Multiple executor support
Date Sun, 08 Nov 2015 19:01:07 GMT
Read "all" as "most" for available execution parameters for the executor to act on. Few like
the user can be controlled from operator like you mention. 


Thx


Sent from my iPhone

On Nov 8, 2015, at 10:50 AM, Bill Farner <wfarner@apache.org> wrote:

>> all standard mesos inputs are available to the custom executor which is driven from
client
> 
> This is the part i'm not convinced is a good idea due to security issues.  The unix user
of the executor (via setuid) is set at the framework level [1].  This would force you to use
a completely unprivileged user for the whole cluster to avoid escalation issues.
> 
> Perhaps further down the line we could create an admin API for adding/removing executors,
however.
> 
> [1] https://github.com/apache/mesos/blob/1e7be5ca417871828e21c38de07142ee9b2c772e/include/mesos/mesos.proto#L208-L212
> 
>> On Sun, Nov 8, 2015 at 10:18 AM,  <meghdoot_b@yahoo.com> wrote:
>> I see. That would be excellent Bill if you are willing to take on this and actively
drive it. 
>> 
>> As you mention if all standard mesos inputs are available to the custom executor
which is driven from client that would be great. URI set for files in sandbox, data blob (could
be anything that an executor wants from client), set of dynamic ports send as resources to
executor from scheduler as indicated by client (some convention defined in client request
that is understood by scheduler), env variables etc that I can think of.
>> 
>> Thx
>> 
>> Sent from my iPhone
>> 
>> On Nov 7, 2015, at 7:39 PM, Bill Farner <wfarner@apache.org> wrote:
>> 
>>>> Could you kindly elaborate with an example how the end user can pass in the
executor of their choice in the API . I remember that other than a name passed on the API
there was no scope to pass in the path of the custom executor or things like Fetcher URI sets
that can be downloaded by mesos in the sandbox.
>>> 
>>> The user submitting jobs cannot do this, but the cluster administrator can specify
an arbitrary executor for the cluster (which is effectively done today by setting the URI
of the thermos executor).  In other words, i can use a non-thermos executor by using the scheduler
thrift API so long as the executor data field is compatible with the executor referenced by
the scheduler command line.
>>> 
>>> At any rate, i think adding support for custom executor fields (mirroring mesos
protobuf fields) is valuable.  I would like to start there, and at that point adding support
for multiple executors becomes trivial.
>>> 
>>> I'm happy to take this work on myself, though it will probably be another week
or two before patches start landing.  I already have a patch locally that does a bunch of
necessary prep work, however.
>>> 
>>> Thanks for pushing on this feature!
>>> 
>>> 
>>>> On Wed, Nov 4, 2015 at 8:50 AM, <meghdoot_b@yahoo.com> wrote:
>>>> Could you kindly elaborate with an example how the end user can pass in the
executor of their choice in the API . I remember that other than a name passed on the API
there was no scope to pass in the path of the custom executor or things like Fetcher URI sets
that can be downloaded by mesos in the sandbox.
>>>> I remember we discussed to make the change like marathon where one can pass
in all details of the executor from client but you had opposed citing security. That should
be captured in the ticket. Hence, the patch was made to load up selected executors from the
scheduler side and depending on the executor name passed in the API the relevant executor
configs can be used and filled in task.
>>>> 
>>>> So kind of confused. Examples will help.
>>>> 
>>>> Thx
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On Nov 4, 2015, at 8:17 AM, Bill Farner <wfarner@apache.org> wrote:
>>>>> 
>>>>> My change in stance partially came from the realization that the scheduler
is not currently coupled to the executor implementation.  As a result, an API consumer (bypassing
the client) could already use any executor implementation they choose.  This could be interpreted
as custom executor support, and i would like to know if that satisfies the general use case.
>>>>> 
>>>>> I am slightly uneasy about supporting user-selected executors on the
scheduler simply because of the complexity that comes with it (the configuration complexity
is one, but there will also likely be a need for ACLs, and even greater complexity in the
client to support multiple/arbitrary configuration input formats).
>>>>> 
>>>>> So, my question becomes - is this (end-user-selected executors) a real
use case today, or a hypothetical one?  If it a real use case, some elaboration would be useful
to make sure we are building the right software for it.
>>>>> 
>>>>>> On Tue, Nov 3, 2015 at 9:30 PM, <meghdoot_b@yahoo.com> wrote:
>>>>>> Bill what do you think?
>>>>>> 
>>>>>> Thx
>>>>>> 
>>>>>> Sent from my iPhone
>>>>>> 
>>>>>> > On Nov 2, 2015, at 1:00 PM, Zameer Manji <zmanji@apache.org>
wrote:
>>>>>> >
>>>>>> > +wfarner
>>>>>> >
>>>>>> > I believe Bill was heavily involved in reviewing the proposed
patch and
>>>>>> > design. Bill, care to comment on what you think here?
>>>>>> >
>>>>>> >> On Mon, Nov 2, 2015 at 12:55 PM, <meghdoot_b@yahoo.com.invalid>
wrote:
>>>>>> >>
>>>>>> >> Do we have a decision on this?
>>>>>> >>
>>>>>> >> https://issues.apache.org/jira/plugins/servlet/mobile#issue/AURORA-1376
>>>>>> >>
>>>>>> >> It would help to know where we stand on this.
>>>>>> >>
>>>>>> >> Thx
>>>>>> >>
>>>>>> >>
>>>>>> >> Sent from my iPhone
>>>>>> >>
>>>>>> >> --
>>>>>> >> Zameer Manji
>>>>>> >>
>>>>>> >>
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message