aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From meghdoo...@yahoo.com.INVALID
Subject Re: Multiple executor support
Date Sun, 08 Nov 2015 18:18:24 GMT
I see. That would be excellent Bill if you are willing to take on this and actively drive it.


As you mention if all standard mesos inputs are available to the custom executor which is
driven from client that would be great. URI set for files in sandbox, data blob (could be
anything that an executor wants from client), set of dynamic ports send as resources to executor
from scheduler as indicated by client (some convention defined in client request that is understood
by scheduler), env variables etc that I can think of.

Thx

Sent from my iPhone

On Nov 7, 2015, at 7:39 PM, Bill Farner <wfarner@apache.org> wrote:

>> Could you kindly elaborate with an example how the end user can pass in the executor
of their choice in the API . I remember that other than a name passed on the API there was
no scope to pass in the path of the custom executor or things like Fetcher URI sets that can
be downloaded by mesos in the sandbox.
> 
> The user submitting jobs cannot do this, but the cluster administrator can specify an
arbitrary executor for the cluster (which is effectively done today by setting the URI of
the thermos executor).  In other words, i can use a non-thermos executor by using the scheduler
thrift API so long as the executor data field is compatible with the executor referenced by
the scheduler command line.
> 
> At any rate, i think adding support for custom executor fields (mirroring mesos protobuf
fields) is valuable.  I would like to start there, and at that point adding support for multiple
executors becomes trivial.
> 
> I'm happy to take this work on myself, though it will probably be another week or two
before patches start landing.  I already have a patch locally that does a bunch of necessary
prep work, however.
> 
> Thanks for pushing on this feature!
> 
> 
>> On Wed, Nov 4, 2015 at 8:50 AM, <meghdoot_b@yahoo.com> wrote:
>> Could you kindly elaborate with an example how the end user can pass in the executor
of their choice in the API . I remember that other than a name passed on the API there was
no scope to pass in the path of the custom executor or things like Fetcher URI sets that can
be downloaded by mesos in the sandbox.
>> I remember we discussed to make the change like marathon where one can pass in all
details of the executor from client but you had opposed citing security. That should be captured
in the ticket. Hence, the patch was made to load up selected executors from the scheduler
side and depending on the executor name passed in the API the relevant executor configs can
be used and filled in task.
>> 
>> So kind of confused. Examples will help.
>> 
>> Thx
>> 
>> Sent from my iPhone
>> 
>>> On Nov 4, 2015, at 8:17 AM, Bill Farner <wfarner@apache.org> wrote:
>>> 
>>> My change in stance partially came from the realization that the scheduler is
not currently coupled to the executor implementation.  As a result, an API consumer (bypassing
the client) could already use any executor implementation they choose.  This could be interpreted
as custom executor support, and i would like to know if that satisfies the general use case.
>>> 
>>> I am slightly uneasy about supporting user-selected executors on the scheduler
simply because of the complexity that comes with it (the configuration complexity is one,
but there will also likely be a need for ACLs, and even greater complexity in the client to
support multiple/arbitrary configuration input formats).
>>> 
>>> So, my question becomes - is this (end-user-selected executors) a real use case
today, or a hypothetical one?  If it a real use case, some elaboration would be useful to
make sure we are building the right software for it.
>>> 
>>>> On Tue, Nov 3, 2015 at 9:30 PM, <meghdoot_b@yahoo.com> wrote:
>>>> Bill what do you think?
>>>> 
>>>> Thx
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>> > On Nov 2, 2015, at 1:00 PM, Zameer Manji <zmanji@apache.org> wrote:
>>>> >
>>>> > +wfarner
>>>> >
>>>> > I believe Bill was heavily involved in reviewing the proposed patch
and
>>>> > design. Bill, care to comment on what you think here?
>>>> >
>>>> >> On Mon, Nov 2, 2015 at 12:55 PM, <meghdoot_b@yahoo.com.invalid>
wrote:
>>>> >>
>>>> >> Do we have a decision on this?
>>>> >>
>>>> >> https://issues.apache.org/jira/plugins/servlet/mobile#issue/AURORA-1376
>>>> >>
>>>> >> It would help to know where we stand on this.
>>>> >>
>>>> >> Thx
>>>> >>
>>>> >>
>>>> >> Sent from my iPhone
>>>> >>
>>>> >> --
>>>> >> Zameer Manji
>>>> >>
>>>> >>
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message