airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siddharth anand <san...@apache.org>
Subject Re: Setting Connections via Env Vars
Date Sat, 01 Oct 2016 06:47:29 GMT
I see. Thanks for the explanation.

Then, wouldn't you like Airflow's connections views to read, edit, and
delete connection information, whether that connection information is
stored in the db or in the OS environment? Currently, OS env-provided
connections are not exposed by the web app! Sure, they can be accessed by
the hooks, but the connection list/edit views don't expose them. Do you
find this a short-coming?

-s

On Fri, Sep 30, 2016 at 10:07 PM, George Leslie-Waksman <
george@cloverhealth.com> wrote:

> We (Clover Health) currently deploy airflow to a 12-factor style
> environment (Aptible) and exclusively use environment variables to define
> connections. Removal of this feature would require a significant refactor
> of how we use Airflow and would likely force us away from the mainline code
> branch.
>
> I would be happy to discuss things further, if you need more details.
> Certainly we could work around, and certainly there are things we don't
> like about the current system, but we only have so many places that we can
> devote resources at any given point in time.
>
> --George Leslie-Waksman (Clover Health)
>
>
> On Fri, Sep 30, 2016 at 9:20 PM siddharth anand <sanand@apache.org> wrote:
>
> I'm trying to understand the impetus for a feature - the ability to define
> connections using OS environment vars.
>
> https://airflow.incubator.apache.org/concepts.html#connections
>
> It seems folks use this so that they can set up Airflow connections using
> automation (e.g. Ansible, chef, puppet. etc...).
>
> I've never used this feature and would prefer to have CLI commands set up
> connections (e.g. https://github.com/apache/incubator-airflow/pull/1802).
>
> It does not seem to me that these connections are revealed in the
> connection list view page or imported into the db table. It seems like the
> OS environment serves as a parallel repository for connection information,
> side-by-side the airflow metadata db. I don't like this this "parallel"
> repository as it is one more source of metadata to keep track of.
>
> If anyone is using the connections being set via env_var for any purpose
> aside from what I've listed here, please make it known. Else, I would like
> to add this feature to the list of 2.0 deprecated features.
>
> In the meantime, I will spend some time to review and push the CLI
> connection PR forward. We currently already support setting Variables and
> Pools via the CLI. It makes sense to manage Connections in the same way.
> The CLI can be called via automation toolsets like those listed above.
>
> -s
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message