airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <fo...@driesprong.frl>
Subject Re: Airflow with Celery
Date Wed, 16 May 2018 08:56:45 GMT
I had similar issues with Airflow running the Celery executor.

The celery_result_backend should be a persistent database like Postgres or
MySql. What broker are you using? I would recommend using Redis or
RabbitMQ, depending on what you like the most.

Cheers, Fokko

2018-05-15 21:12 GMT+02:00 David Capwell <dcapwell@gmail.com>:

> What I find is that when celery rejects we hit this.  For us we don't do
> work on the hosts so solve by over provisioning tasks in celery
>
> On Tue, May 15, 2018, 6:30 AM Andy Cooper <andy.cooper@astronomer.io>
> wrote:
>
>> I have had very similar issues when there was a problem with the
>> connection
>> string pointing to the message broker. Triple check those connection
>> strings and attempt to connect outside of airflow.
>>
>> On Tue, May 15, 2018 at 9:27 AM Goutham Pratapa <pratapagoutham@gmail.com
>> >
>> wrote:
>>
>> > Hi all,
>> >
>> > I have been using airflow with Celery executor in the background
>> >
>> > https://hastebin.com/sipecovomi.ini --> airflow.cfg
>> >
>> > https://hastebin.com/urutokuvoq.py   --> The dag I have been using
>> >
>> >
>> >
>> > This shows that the dag is always in running state.
>> >
>> >
>> >
>> >
>> > Airflow flower shows nothing in the tasks or in the broker.
>> >
>> >
>> > Did I miss anything can anyone help me in this regard.
>> >
>> >
>> > --
>> > Cheers !!!
>> > Goutham Pratapa
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message