airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Brard <emmanuel.br...@getyourguide.com>
Subject Re: Task Getting Stuck in Queued State
Date Wed, 29 May 2019 06:49:38 GMT
Hi,

We have noticed the same thing. On way for us to unblock it was to restart
the scheduler (there is a function that check missing tasks in executor at
the scheduler start-up phase).
We implemented a health-check that checks for these kind of stuck queued
tasks, but obviously fixing it in airflow core would be better.

Cheers,
E

On Wed, May 29, 2019 at 8:01 AM ramandumcs@gmail.com <ramandumcs@gmail.com>
wrote:

> Hi All,
> We are observing intermittently  that Tasks get stuck in queued state and
> never get executed by Airflow.
> On debugging it we found that one of the queued dependency was not met due
> to which task did not move from queued to running state. So task remained
> in queued state.
> (are_dependencies_met function returned false for QUEUE_DEPS inside
> _check_and_change_state_before_execution)
>
> By looking into scheduler code it seems that scheduler does not reschedule
> the queued state tasks due to which task never got added to executor queue
> again and remained stuck in queued state.
>
> There is a logic inside _check_and_change_state_before_execution function
> to move the task from queued to None state(which gets picked by scheduler
> for rescheduling) if RUN_DEPS are not met but this logic seems to be
> missing for QUEUE_DEPS.
>
> It seems that task should be moved to None state even if QUEUE_DEPS are
> not met.
> Any help/pointer on this would be useful.
>
> Thanks,
> Raman Gupta
>
>
>

-- 








GetYourGuide AG

Stampfenbachstrasse 48  

8006 Zürich

Switzerland



 <https://www.facebook.com/GetYourGuide>  
<https://twitter.com/GetYourGuide>  
<https://www.instagram.com/getyourguide/>  
<https://www.linkedin.com/company/getyourguide-ag>  
<http://www.getyourguide.com>








Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message