airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Davydov <dan.davy...@airbnb.com.INVALID>
Subject Re: Good practices around restarting scheduler and server of airflow
Date Thu, 13 Jul 2017 20:58:31 GMT
Airbnb switched to running the scheduler without restarts about a month
ago, and it is working well. The one issue you might hit is if you are
using a version of Celery with a bug (where workers will pull off tasks
even if they don't have space to run them), tasks can potentially be lost
once they are sent to your Celery backend. With restarts these tasks are
automatically resent.

On Thu, Jul 13, 2017 at 1:38 PM, manish ranjan <cse1.manish@gmail.com>
wrote:

> Hi All,
>
> We have recently started to use airflow to make our data engineering more
> robust.
> 1. I wanted to know if there are good practices around scheduling a daily/
> bi-weekly/ weekly restart of the airflow server and scheduler? We are using
> just the instance and not cluster.
> 2.  Also wanted to know the troubles I may run into if someone is ready to
> share based on experience?
>
> ~Manish
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message