Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
References: <CAB=riaCB7ZGV_dK+Ot4OHZ=Vkro8RjArpctOGCE7eCP_Wb6A=Q@mail.gmail.com>
In-Reply-To: <CAB=riaCB7ZGV_dK+Ot4OHZ=Vkro8RjArpctOGCE7eCP_Wb6A=Q@mail.gmail.com>
From: George Leslie-Waksman <george@cloverhealth.com.INVALID>
Date: Fri, 04 Aug 2017 17:21:39 +0000
Message-ID: <CANcybpBzf1UFLwkkW=skRZMEW2SeMv7VoZnv=oOd3cxKjXZxDw@mail.gmail.com>
Subject: Re: Tasks stay queued when they fail in celery
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary="001a1134e9a089e7500555f0bb42"
archived-at: Fri, 04 Aug 2017 17:22:02 -0000

--001a1134e9a089e7500555f0bb42
Content-Type: text/plain; charset="UTF-8"

We've seen this before as well, it's a bug in the Celery Executor that has
a bunch of different manifestations.

There is at least one open issue relating to this bug:
https://issues.apache.org/jira/browse/AIRFLOW-1463

I have been working on a fix but it's likely to be a few more days before I
have a chance to make some progress.

--George

On Fri, Jul 28, 2017 at 5:05 PM David Capwell <dcapwell@gmail.com> wrote:

> We noticed that in the past few days we keep seeing tasks stay in the
> queued state.  Looking into celery, we see that the task had failed.
>
> Traceback (most recent call last):
>   File "/python/lib/python2.7/site-packages/celery/app/trace.py", line
> 367, in trace_task
>     R = retval = fun(*args, **kwargs)
>   File "/python/lib/python2.7/site-packages/celery/app/trace.py", line
> 622, in __protected_call__
>     return self.run(*args, **kwargs)
>   File
> "/python/lib/python2.7/site-packages/airflow/executors/celery_executor.py",
> line 59, in execute_command
>     raise AirflowException('Celery command failed')
> AirflowException: Celery command failed
>
>
> Why does airflow not learn about this and recover? And what can we do to
> prevent this?
>
> Thanks for your time reading this email.
>

--001a1134e9a089e7500555f0bb42--