airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Imberman <daniel.imber...@gmail.com>
Subject Re: Dynamic Airflow config reloads
Date Fri, 08 Dec 2017 15:07:06 GMT
At least for the k8s executor it would be easier to just restart the
scheduler pod. I can see some potential issues if some configs are stored
in local variables.
On Wed, Dec 6, 2017 at 11:45 AM Feng Lu <fenglu@google.com.invalid> wrote:

> Hi,
>
> It's probably well-known that Airflow only loads config file (i.e.,
> airflow.cfg) at instance creation time, if one needs to change the config
> file, all Airflow instances have to be restarted (understand that Airflow
> worker does restart itself for each task execution and therefore picks up
> the latest config updates).
>
> Are people from this mailing group interested in adding dynamic config
> loading support inside AirflowConfigParser (
>
> https://github.com/apache/incubator-airflow/blob/master/airflow/configuration.py#L110
> )?
>
> Initial implementation ideas:
> - introduce threadling.Rlock to AirflowConfigParser and guard all method
> access
> - add a periodical timer task that reads in the config file (of course
> needs to acquire the lock beforehand).
>
> Since config data is always accessed via the AirflowConfigParser object,
> this essentially gives us dynamic config update without restarting Airflow
> scheduler/webserver.
>
> Thoughts?
> Thank you.
>
> Feng
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message