Mailing-List: contact dev-help@aurora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.apache.org
Received-SPF: Pass (protection.outlook.com: domain of cern.ch designates
 188.184.36.50 as permitted sender) receiver=protection.outlook.com;
 client-ip=188.184.36.50; helo=CERNMX11.cern.ch;
Subject: Re: Questions about Aurora scheduling policy
To: <dev@aurora.apache.org>
References: <5653BEAD.1060005@cern.ch>
 <CAFWq12VEK4mBj1bpgyDmLZ_NO2CKNoEGyE3mwiMwvXcbiixGMA@mail.gmail.com>
From: Riccardo Poggi <riccardo.poggi@cern.ch>
Message-ID: <5654C539.7050004@cern.ch>
Date: Tue, 24 Nov 2015 21:14:49 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: 
 <CAFWq12VEK4mBj1bpgyDmLZ_NO2CKNoEGyE3mwiMwvXcbiixGMA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:23
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Nov 2015 20:14:50.7857
 (UTC)
X-MS-Exchange-CrossTenant-Id: c80d3499-4a40-4a8c-986e-abce017d6b19
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: 
 TenantId=c80d3499-4a40-4a8c-986e-abce017d6b19;Ip=[188.184.36.50];Helo=[CERNMX11.cern.ch]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR06MB1412

Thanks Bill,

> I will answer your questions directly, but it may
> also make sense to have a higher-level discussion about your requirements
> to possibly offer alternative approaches.
Sure, that sounds very interesting.

The main scheduling constraint in our system is that it cannot tolerate 
an undefined pending period, it needs at least some fast-feedback.

Example scenario: the system asks to launch N instances of a given 
process with a specified set of resource constraints, now the scheduler 
should either
- start them all successfully
- start only a part of them
     + because of resource saturation for example
- fail to start them
     + because those resources are not at all available in the farm, or 
they are effectively busy, or ...
without queuing, but just returning the operation result.


There are also other, more functional, requirements. The most 
interesting ones probably are:
* Hooks. For good integration with the infrastructure, like publishing 
of information, access management control, and so on...
* Fault tolerance and Error recovery. A failure in either the scheduler 
or executor shall not take down the managed processes and shall recover 
it after a restart for example.
* Ownership. Possibility to set uid owner of the underlying processes.
* Notification system. Provide informations about the jobs/processes 
status (both in pull or push mode)


> 3. Would it possible to have Aurora handle tasks with no resource
>> constraints?
> [...] but the real question is:
> what behavior do you want from scheduling without accounting?
Yes, that is indeed the question. What would be the default scheduling 
behaviour in case of resource abundance?
Would it be possible to impose a "spreading" constraint on a subset of 
the farm that a-priori I know is able to handle a defined set of jobs?


Cheers,
    Riccardo


On 11/24/2015 06:38 AM, Bill Farner wrote:
> Welcome, happy to help!  I will answer your questions directly, but it may
> also make sense to have a higher-level discussion about your requirements
> to possibly offer alternative approaches.
>
> 1. Would it possible to subscribe to state change for a given job/task and
>> receive notifications?
>
> Not today, but i'm very open to the idea.  I think there are some cool
> things you could implement with this behavior.  A concern i often have,
> though, is that consumers of this data cannot handle cases where an event
> fails to be delivered (i.e. they want a replica of the scheduler's state).
> At any rate, i'd love to offer this behavior!
>
> 2. Would it possible to set a Pending time-out for tasks that take too long
>> to be Assigned?
>
> Not currently.  You could implement this by polling the API and killing
> tasks that took too long to schedule.  This would allow you to decide how
> to react (if at all).
>
> 3. Would it possible to have Aurora handle tasks with no resource
>> constraints?
>
> No.  Both Aurora and Mesos require CPU and memory to be specified for
> tasks.  A client of Aurora could choose defaults, but the real question is:
> what behavior do you want from scheduling without accounting?
>
>
> On Mon, Nov 23, 2015 at 5:34 PM, Riccardo Poggi <riccardo.poggi@cern.ch>
> wrote:
>
>> Hello,
>>
>> In order to introduce Aurora into our distributed system we would like to
>> have it slowly, and hopefully transparently, replace what is currently the
>> process manger component. To do that it would have to programmatically
>> interface with other parts of the system that are, at the moment, taking
>> care of what it could be considered the "active" orchestration.
>>
>> I've tried Aurora and looked at the docs, but I'm still left with some
>> open questions:
>>
>> 1. Would it possible to subscribe to state change for a given job/task and
>> receive notifications?
>>
>> 2. Would it possible to set a Pending time-out for tasks that take too
>> long to be Assigned?
>>
>> 3. Would it possible to have Aurora handle tasks with no resource
>> constraints?
>>
>>
>> Thanks,
>>    Riccardo
>>