airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <>
Subject Re: [DISCUSS] Managing provider Connections via UI in managed Airflow services
Date Mon, 14 Jun 2021 16:28:46 GMT
Can you elaborate (privately if you have to) on what the security concerns are? Since as I
understand it the web server is powery deployment, so anything should be limited to one customer/user/deployment.

There is also the new "test connection" feature that will need the provider code installed
to work.

Then there's the issue of third party connections - of which there is only going to be more
of over time.


On 14 June 2021 16:35:42 BST, Eugen Kosteev <> wrote:
>Hi Jarek.
>Thanks for the discussion.
>The issue with Connections management in the web server that you described
>is indeed affected Cloud Composer in the released preview image versions of
>Airflow 2.0.1 (link to public issue
> And as you stated, we do
>not install pypi packages in web server image mostly because of security
>As a temporary workaround we baked all connections (list of them with their
>widgets pickled and stored inside) into a web server image, so that
>customers can add/edit them (even though not all providers packages are
>pre-installed). This is a temporary workaround that we came up with for now
>and we are looking for a long-term solution.
>Our thoughts/ideas for alternative solutions:
>1. We do not want to pre-install all providers packages as to not generate
>unnecessary python dependencies. Or maybe we could do this only for web
>server images (not scheduler/worker) but then it is not clear if this is a
>good idea to have such occured discrepancy between pypi dependencies in web
>server vs scheduler/worker images.
>2. Downloading and backing in providers packages (wheel files) into docker
>image and installing customer specific/required version on demand looks
>infeasible, taking into account number of providers, their versions and
>their dependencies.
>- Eugene
>On Sun, Jun 13, 2021 at 6:46 PM Jarek Potiuk <> wrote:
>> Dear Airflow community,
>> Here is another result of discussions. I would like to raise an attention
>> to potential Connection management problems that might affect managed
>> services for Airflow 2.0 and some providers.
>> With Airflow 2.0, connection UI "customisations" are baked into the
>> provider package and in order to see - for example Postgres connection in
>> the UI, you need to have the "postgres" provider installed in the Webserver.
>> As far as I know some of the Managed Airflow services (MWAA, Composer,
>> possibly other) do not currently allow their users installation of
>> additional packages in the webserver (the webserver container is different
>> than the scheduler/worker). This makes it impossible to configure/edit
>> provider connections via UI (unless those providers are pre-installed in
>> the webserver image).
>> While this is understandable from security point of view to forbid "any''
>> package installation, I think the official
>> "apache-airlfow-providers-*" should be allowlisted for those images and
>> installed or otherwise made available (for example via pre-installing all
>> providers in the webserver image if this is not possible from security
>> point of view to rebuild the image dynamically)
>> I wonder what people (and especially the people from MWAA, Composer team)
>> think about it - do I get it right about the security concerns? Any other
>> comments?
>> J.
>> --
>> +48 660 796 129

View raw message