airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sai Phanindhra <phani8...@gmail.com>
Subject Re: Moving Airflow Config to Database.
Date Thu, 15 Nov 2018 12:52:53 GMT
Some of env variables are not getting reflected in supervisord
environment(sometimes new variables are not available, sometimes changes to
existing variables are not reflected)

On Thu 15 Nov, 2018, 17:37 Ash Berlin-Taylor <ash@apache.org wrote:

> > problem with
> > this approach is these env variables wont behave correctly when we
> > subshells
>
>
> Can you explain what you mean by this?
>
> -ash
>
>
> > On 15 Nov 2018, at 12:03, Sai Phanindhra <phani8996@gmail.com> wrote:
> >
> > Hi deng,
> > I am currently using env variables for few airflow config variables which
> > may differ across machines(airflow folder, log folder etc..,) Problem
> with
> > this approach is these env variables wont behave correctly when we
> > subshells. ( I faced issues when i added airflow jobs in supervisord).
> >  Moving config to db not only addresses this issue, it will give
> provision
> > to change config from UI.(most of the time, we cant give box access to
> all
> > users. ). Config is db makes it easy to create/update config without
> > touching code.
> >
> > On Thu 15 Nov, 2018, 16:04 Deng Xiaodong <xd.deng.r@gmail.com wrote:
> >
> >> A few solutions that may address your problem:
> >>
> >> - Specify your configurations in environment variables, so it becomes
> much
> >> easier to manage across machines
> >> - use network attached storage to save your configuration file and
> mount it
> >> to all your machines (this can address DAG file sync as well)
> >> - ...
> >>
> >> Personally I don’t see point moving configuration to DB.
> >>
> >>
> >> XD
> >>
> >> On Thu, Nov 15, 2018 at 18:29 Sai Phanindhra <phani8996@gmail.com>
> wrote:
> >>
> >>> Hello Airflow users,
> >>>      I recently encountered as issue with airflow. I am maintaining a
> >>> airflow cluster, whenever i make a change in airflow configuration in
> one
> >>> of the machine i have to consciously copy these changes to other
> machines
> >>> in airflow cluster. Problem with this is it's a manual process and
> >>> something users tend to forget to sync changes. I want move airflow
> >> config
> >>> to a database. I will be happy if you can share your valuable
> >>> inputs/thoughts on this.
> >>>
> >>> --
> >>> Sai Phanindhra,
> >>> Ph: +91 9043258999
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message