airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Viraj Parekh <vi...@astronomer.io>
Subject Re: Celery conflicts
Date Thu, 25 Apr 2019 20:51:21 GMT
We used to run into that error a ton.

If I remember correctly, this
<https://github.com/apache/airflow/commit/8f15661001706e5bd76a7134c4c8ece06ad83a63#diff-54a57ccc2c8e73d12c812798bf79ccb2>
commit,
along with adding memory to the worker node ended up fixing it!

Viraj

On Thu, Apr 25, 2019 at 4:48 PM Bolke de Bruin <bdbruin@gmail.com> wrote:

> What version of Airflow?
>
>
> On 25 April 2019 at 22:32:25, Teresa Martyny (
> teresa.martyny@omadahealth.com)
> wrote:
>
> Hi all,
>
> We recently doubled the number of tasks and dags in our core data pipeline
> in Airflow (~3500 tasks now). We started doing a few things to handle the
> increased load: Increased resources server side to handle the increased CPU
> usage, Added priority_weight to tasks, Switched to Celery/Flower from
> LocalExecutor.
>
> We started seeing some interesting errors associated with Celery no longer
> running tasks. We see the tasks in the Airflow UI sitting in the queue in a
> scheduled or None state. Flower shows celery processing nothing. It starts
> around 45-60mins into the run.
>
> The errors are along the lines of:
> ERROR - Executor reports task instance <TaskInstance: yadayada 2019-04-24
> 18:00:00+00:00 [queued]> finished (success) although the task says its
> queued. Was the task killed externally?
> NoneType: None
>
> For the same tasks.... we're seeing try_number set to a negative number....
> Any thoughts on what could lead to Airflow assigning a negative number to
> try_number? Any thoughts on what results in the executor and the task being
> out of sync with the task state? Are these related?
>
> Any ideas would be welcome. The tasks themselves in the Airflow UI have no
> logs showing when I go to that view... so it's sad.
>
> Thanks!
> *Teresa Martyny*
> pronouns: she, her, hers
> Software Engineer | Data Team Lead | Omada Health
> <https://www.omadahealth.com/>
> 500 Sansome St #200, SF, CA 94111
>
> *What is Omada?* <https://vimeo.com/203386025>
>
> --
> This email may contain material that is confidential and/or privileged for
> the sole use of the intended recipient. Any review, reliance, or
> distribution by others or forwarding without express permission is strictly
> prohibited. If you are not the intended recipient, please contact the
> sender and delete all copies. Also note that email is not an appropriate
> way to send protected health information to Omada Health employees. Please
> use your discretion when responding to this email.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message