airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Viraj Parekh <>
Subject Re: Celery conflicts
Date Thu, 25 Apr 2019 20:51:21 GMT
We used to run into that error a ton.

If I remember correctly, this
along with adding memory to the worker node ended up fixing it!


On Thu, Apr 25, 2019 at 4:48 PM Bolke de Bruin <> wrote:

> What version of Airflow?
> On 25 April 2019 at 22:32:25, Teresa Martyny (
> wrote:
> Hi all,
> We recently doubled the number of tasks and dags in our core data pipeline
> in Airflow (~3500 tasks now). We started doing a few things to handle the
> increased load: Increased resources server side to handle the increased CPU
> usage, Added priority_weight to tasks, Switched to Celery/Flower from
> LocalExecutor.
> We started seeing some interesting errors associated with Celery no longer
> running tasks. We see the tasks in the Airflow UI sitting in the queue in a
> scheduled or None state. Flower shows celery processing nothing. It starts
> around 45-60mins into the run.
> The errors are along the lines of:
> ERROR - Executor reports task instance <TaskInstance: yadayada 2019-04-24
> 18:00:00+00:00 [queued]> finished (success) although the task says its
> queued. Was the task killed externally?
> NoneType: None
> For the same tasks.... we're seeing try_number set to a negative number....
> Any thoughts on what could lead to Airflow assigning a negative number to
> try_number? Any thoughts on what results in the executor and the task being
> out of sync with the task state? Are these related?
> Any ideas would be welcome. The tasks themselves in the Airflow UI have no
> logs showing when I go to that view... so it's sad.
> Thanks!
> *Teresa Martyny*
> pronouns: she, her, hers
> Software Engineer | Data Team Lead | Omada Health
> <>
> 500 Sansome St #200, SF, CA 94111
> *What is Omada?* <>
> --
> This email may contain material that is confidential and/or privileged for
> the sole use of the intended recipient. Any review, reliance, or
> distribution by others or forwarding without express permission is strictly
> prohibited. If you are not the intended recipient, please contact the
> sender and delete all copies. Also note that email is not an appropriate
> way to send protected health information to Omada Health employees. Please
> use your discretion when responding to this email.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message