airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Desiree Cox <d...@zendesk.com.INVALID>
Subject Re: Is anybody using CeleryExecutor + redis-sentinel message broker?
Date Wed, 08 Mar 2017 21:20:55 GMT
Yes it does help. Thanks Robin! I struggled a little on account of having
to pass this parameter as a dict but I figured it out and I can copy what
you did to create the JIRA ticket and PR. Thanks again!

On Wed, Mar 8, 2017 at 1:15 AM, Miller, Robin <
Robin.Miller@affiliate.oliverwyman.com> wrote:

> Hi Desiree,
>
>
> It sounds to me like you'll probably need to add an additional
> configuration option to pass to Celery. In the file
> airflow/executors/celery_executor.py you can see all of the config
> options that are passed into Airflow in the CeleryConfig class. I've
> currently got a PR open for a change of this nature at the moment:
> https://github.com/apache/incubator-airflow/pull/1912/files
> <https://github.com/apache/incubator-airflow/pull/1912/files>.
>
>
> Hope this helps,
>
> Robin Miller
> OLIVER WYMAN
> robin.miller@affiliate.oliverwyman.com<mailto:robin.
> miller@affiliate.oliverwyman.com>
> www.oliverwyman.com<http://www.oliverwyman.com/
> <http://www.oliverwyman.com/>>
>
> ________________________________
> From: Desiree Cox <dcox@zendesk.com.INVALID>
> Sent: 07 March 2017 18:22:03
> To: dev@airflow.incubator.apache.org
> Subject: Is anybody using CeleryExecutor + redis-sentinel message broker?
>
>
> Hello! I am a very new engineer at my company and I have been asked to set
> up Airflow with Celery. We have existing Redis resources for the message
> broker, but I am not allowed to connect to redis directly, I must do it
> through redis-sentinel. Celery 4 can use redis-sentinel but I don't see how
> to pass the configuration through Airflow—you have to send it a dictionary
> called 'broker_transport_options' with the name of the Redis service you
> are looking for. Has anybody gotten the CeleryExecutor working with
> redis-sentinel? My teammate says I should contribute a PR to Airflow (which
> I am happy to try) but I want to check in case anybody has suggestions.
> Thank you!
>
> --
> Desiree Cox
> Associate Software Engineer
> Zendesk
>
> ________________________________
> This e-mail and any attachments may be confidential or legally privileged.
> If you received this message in error or are not the intended recipient,
> you should destroy the e-mail message and any attachments or copies, and
> you are prohibited from retaining, distributing, disclosing or using any
> information contained herein. Please inform us of the erroneous delivery by
> return e-mail. Thank you for your cooperation.
>



-- 
Desiree Cox
Associate Software Engineer
Zendesk

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message