airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrii Kinash <and...@adsquare.com>
Subject Airflow 1.8 CeleryExecutor: multiple servers in broker_url
Date Wed, 17 Jan 2018 10:51:49 GMT
Dear Airflow Community,

I was able to successfully setup my Celery executor pointing to one RabbitMQ server (broker).
 However when I’m trying to setup it to point to multiple RabbitMQ servers for HA – it
doesn’t seem to work, even thought I’m doing it exactly as it’s written in the Celery
documentation: http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_url
<http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_url>


When I’m using:

> broker_url = [
>                 'pyamqp://usr:pwd@host1:5672//airflow1',
>                 'pyamqp://usr:pwd@host2:5672//airflow1'
> ]


I’m getting:

> Jan 17 10:19:58 apph05 airflow[14606]: ConfigParser.ParsingError: File contains parsing
errors: /…./airflow.cfg
> Jan 17 10:19:58 apph05 airflow[14606]: [line 162]: ']\n’


and when 

> broker_url = 'pyamqp://usr:pwd@host1:5672//airflow1;pyamqp://usr:pwd@host2:5672//airflow1

getting:

> Jan 17 10:22:44 apph05 airflow[15336]: File “/…./airflow/usr/local/lib/python2.7/dist-packages/kombu/transport/__init__.py",
line 62, in resolve_transport
> Jan 17 10:22:44 apph05 airflow[15336]: raise KeyError('No such transport: {0}'.format(transport))


Also I tried different approaches, with or without quotes etc., but the end result was the
same. I’m not really sure whether it’s a bug or am I doing something wrong?

Thank you,
Andrii
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message