airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: Task Priority per schedule run or applied on entire queue
Date Wed, 08 Mar 2017 04:48:01 GMT
It prioritizes before sending in the message queue. This assume that there
are tasks that cannot be scheduled because of dependency limitations as in
Airflow pools are full or max_concurrency is reached. Once the task is in
the message queue (Celery) Airflow has no more control over it.

Max

On Tue, Mar 7, 2017 at 7:01 PM, Amit Jain <aj2011it@gmail.com> wrote:

> Please provide your input to this misconception.
>
> On Mar 6, 2017 11:46 PM, "Amit Jain" <aj2011it@gmail.com> wrote:
>
> > Hi All,
> >
> > I have a doubt related to task instance priority. When we specify the
> > priority_weight on the operator, does task priority re-ordering happens
> > between the eligible task instances at the time of Scheduler run or does
> it
> > re-order queue in AMQP supported broker (RabbitMQ supports priority_queue
> > 3.5.0 onwards)?
> >
> > I think first case is happening here. Here is code from the master
> branch.
> >
> > // jobs.py
> > def _execute_task_instances(....):
> >
> > priority_sorted_task_instances = sorted(
> >                 task_instances, key=lambda ti: (-ti.priority_weight,
> > ti.execution_date))
> >
> >
> >
> > --
> > Thanks,
> > Amit
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message