airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: possible bug: scheduler dont schedule manual triggered dag on 1.8.b2
Date Wed, 25 Jan 2017 08:32:17 GMT
Probably (havent tested it yet) this makes it work again:

TI = models.TaskInstance
task_instances_to_examine = (
    session
    .query(TI)
    .filter(TI.dag_id.in_(simple_dag_bag.dag_ids))
    .filter(TI.state.in_(states))
    .join(DagRun, and_(TI.dag_id == DagRun.dag_id,
                       TI.execution_date == DagRun.execution_date,
                       DagRun.state == State.RUNNING,
                       DagRun.run_id.notlike('backfill' + '%')))
    .all()
)


Bolke


> On 25 Jan 2017, at 09:00, Bolke de Bruin <bdbruin@gmail.com> wrote:
> 
> Thanks for tracking this down Jayesh. I suspect not wanting ‘backfills’ is the reason
for it (ie. backfills are not scheduled yet), but I need to look through the code. Maybe Dan
or Paul can chime in?
> 
> Bolke
> 
>> On 25 Jan 2017, at 02:46, Jayesh Senjaliya <jhsonline@gmail.com> wrote:
>> 
>> I am running airflow 1.8.b2
>> 
>> a week before I had perfectly running dag which i put it on 1.8.b2 and I
>> find that scheduler is not scheduling the task even though the dag run was
>> created.
>> 
>> I debugged the issue and came down to this piece of code where scheduler
>> finds task to schedule
>> 
>> TI = models.TaskInstance
>> task_instances_to_examine = (
>>   session
>>   .query(TI)
>>   .filter(TI.dag_id.in_(simple_dag_bag.dag_ids))
>>   .filter(TI.state.in_(states))
>>   .join(DagRun, and_(TI.dag_id == DagRun.dag_id,
>>                      TI.execution_date == DagRun.execution_date,
>>                      DagRun.state == State.RUNNING,
>>                      DagRun.run_id.like(DagRun.ID_PREFIX + '%')))
>>   .all()
>> )
>> 
>> 
>> the root cause i think is join in the above query was added recently with
>> this PR  "[AIRFLOW-219][AIRFLOW-398] Cgroups + impersonation"
>> which has condition that run_id should be like DagRun.ID_PREFIX which is
>> "scheduled__" which wont be true for manually triggered dag ( which has
>> run_id like "manual__2017-01-24T17:08:35.251019" )
>> 
>> so I am not sure why we need that condition there at all, does anyone know?
>> 
>> Please comment.
>> 
>> Thanks
>> Jayesh
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message