airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Desiree Cox <>
Subject Re: Is anybody using CeleryExecutor + redis-sentinel message broker?
Date Wed, 08 Mar 2017 21:20:55 GMT
Yes it does help. Thanks Robin! I struggled a little on account of having
to pass this parameter as a dict but I figured it out and I can copy what
you did to create the JIRA ticket and PR. Thanks again!

On Wed, Mar 8, 2017 at 1:15 AM, Miller, Robin <> wrote:

> Hi Desiree,
> It sounds to me like you'll probably need to add an additional
> configuration option to pass to Celery. In the file
> airflow/executors/ you can see all of the config
> options that are passed into Airflow in the CeleryConfig class. I've
> currently got a PR open for a change of this nature at the moment:
> <>.
> Hope this helps,
> Robin Miller
> <>>
> ________________________________
> From: Desiree Cox <>
> Sent: 07 March 2017 18:22:03
> To:
> Subject: Is anybody using CeleryExecutor + redis-sentinel message broker?
> Hello! I am a very new engineer at my company and I have been asked to set
> up Airflow with Celery. We have existing Redis resources for the message
> broker, but I am not allowed to connect to redis directly, I must do it
> through redis-sentinel. Celery 4 can use redis-sentinel but I don't see how
> to pass the configuration through Airflow—you have to send it a dictionary
> called 'broker_transport_options' with the name of the Redis service you
> are looking for. Has anybody gotten the CeleryExecutor working with
> redis-sentinel? My teammate says I should contribute a PR to Airflow (which
> I am happy to try) but I want to check in case anybody has suggestions.
> Thank you!
> --
> Desiree Cox
> Associate Software Engineer
> Zendesk
> ________________________________
> This e-mail and any attachments may be confidential or legally privileged.
> If you received this message in error or are not the intended recipient,
> you should destroy the e-mail message and any attachments or copies, and
> you are prohibited from retaining, distributing, disclosing or using any
> information contained herein. Please inform us of the erroneous delivery by
> return e-mail. Thank you for your cooperation.

Desiree Cox
Associate Software Engineer

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message