airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Georg Walther <georg.walt...@markovian.com>
Subject Re: large number of messages from celeryev pile up in rabbitmq
Date Sun, 18 Jun 2017 13:55:00 GMT
Hi Dima,


do you use RabbitMQ as the Celery result backend?
If so try using e.g. Redis as result backend (parameter
"celery_result_backend" in the airflow.cfg) while
keeping RabbitMQ as the message broker (broker_url).


Best,

Georg

On Mon, May 29, 2017 at 1:59 PM, Dmitry Smirnow <dmitry@yousician.com>
wrote:

> Hi,
>
> I've noticed that in the rabbitmq which is used as a broker for Airflow,
> there are thousands of heartbeat messages from workers piling up (type:
> "worker-heartbeat"). The version I use is 1.7.1.3.
>
> I googled around and it seems that those are the events used by celery
> flower for monitoring.
> I may misunderstood something, but it seemed that to stop those messages I
> should for example set some celery settings to make the unused queues
> expire.
> What would be the right way to deal with it? I'm really not sure which
> config should I touch. Any ideas are welcome and if I need to provide more
> info about configuration - please suggest which one.
>
> Thank you in advance,
> Best regards, Dima
>
> --
>
> Dmitry Smirnov (MSc.)
> Data Engineer @ Yousician
> mobile: +358 50 3015072
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message