airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [DISCUSS] Back to (some) dependency pinning
Date Mon, 24 Jun 2019 13:03:51 GMT
Another suggestion someone (I forget who, sorry) had was that we could maintain a full list
of _fully tested and supported versions_ (i.e. the output of `pip freeze`) - that way people
_can_ use other versions if they want, but we can at least say "use these versions".

I'm not 100% sure how that would work in practice though, but having it be some list we can
update without having to do a release is crucial.

-ash

> On 24 Jun 2019, at 10:00, Jarek Potiuk <Jarek.Potiuk@polidea.com> wrote:
> 
> With the recent Sphinx problem
> <https://issues.apache.org/jira/browse/AIRFLOW-4841>- we got back our
> old-time enemy. In this case sphinx autoapi has been released yesterday to
> 1.1.0 version and it started to caused our master to fail, causing kind of
> emergency rush to fix as master (and all PRs based on it) would be broken.
> 
> I think I have a proposal that can address similar problems without pushing
> us in emergency mode.
> 
> *Context:*
> 
> I wanted to return back to an old discussion - how we can avoid unrelated
> dependencies to cause emergencies on our side where we have to quickly
> solve such dependency issues when they break our builds.
> 
> *Change coming soon:*
> 
> The problems will be partially addressed with last stage of AIP-10 (
> https://github.com/apache/airflow/pull/4938 - pending only Kubernetes test
> fix). It effectively freezes installed dependencies as cached layer of
> docker image for builds which do not touch setup.py - so in case setup.py
> does not change, the dependencies will not be updated to latest ones.
> 
> *Possibly even better long-term solution:*
> 
> I think we should address it a bit better. We had a number of discussions
> on pinning dependencies (for example here
> <https://lists.apache.org/thread.html/9e775d11cce6a3473cbe31908a17d7840072125be2dff020ff59a441@%3Cdev.airflow.apache.org%3E>).
> I think the conclusion there was that airflow is both "library" (for DAGs)
> - where dependencies should not be pinned and end-product (where the
> dependencies should be pinned). So it's a bit catch-22 situation.
> 
> Looking at the problem with Sphinx however It came to me that maybe we can
> use hybrid solution. We pin all the libraries (like Sphinx or Flask) that
> are used to merely build and test the end product but we do not pin the
> libraries (like google-api) which are used in the context of library
> (writing the operators and DAGs).
> 
> What do you think? Maybe that will be the best of both worlds ? Then we
> would have to classify the dependencies and maybe restructure setup.py
> slightly to have an obvious distinction between those two types of
> dependencies.
> 
> J.
> 
> -- 
> 
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
> 
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>


Mime
View raw message