airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <>
Subject Re: [DISCUSS] Back to (some) dependency pinning
Date Mon, 24 Jun 2019 13:03:51 GMT
Another suggestion someone (I forget who, sorry) had was that we could maintain a full list
of _fully tested and supported versions_ (i.e. the output of `pip freeze`) - that way people
_can_ use other versions if they want, but we can at least say "use these versions".

I'm not 100% sure how that would work in practice though, but having it be some list we can
update without having to do a release is crucial.


> On 24 Jun 2019, at 10:00, Jarek Potiuk <> wrote:
> With the recent Sphinx problem
> <>- we got back our
> old-time enemy. In this case sphinx autoapi has been released yesterday to
> 1.1.0 version and it started to caused our master to fail, causing kind of
> emergency rush to fix as master (and all PRs based on it) would be broken.
> I think I have a proposal that can address similar problems without pushing
> us in emergency mode.
> *Context:*
> I wanted to return back to an old discussion - how we can avoid unrelated
> dependencies to cause emergencies on our side where we have to quickly
> solve such dependency issues when they break our builds.
> *Change coming soon:*
> The problems will be partially addressed with last stage of AIP-10 (
> - pending only Kubernetes test
> fix). It effectively freezes installed dependencies as cached layer of
> docker image for builds which do not touch - so in case
> does not change, the dependencies will not be updated to latest ones.
> *Possibly even better long-term solution:*
> I think we should address it a bit better. We had a number of discussions
> on pinning dependencies (for example here
> <>).
> I think the conclusion there was that airflow is both "library" (for DAGs)
> - where dependencies should not be pinned and end-product (where the
> dependencies should be pinned). So it's a bit catch-22 situation.
> Looking at the problem with Sphinx however It came to me that maybe we can
> use hybrid solution. We pin all the libraries (like Sphinx or Flask) that
> are used to merely build and test the end product but we do not pin the
> libraries (like google-api) which are used in the context of library
> (writing the operators and DAGs).
> What do you think? Maybe that will be the best of both worlds ? Then we
> would have to classify the dependencies and maybe restructure
> slightly to have an obvious distinction between those two types of
> dependencies.
> J.
> -- 
> Jarek Potiuk
> Polidea <> | Principal Software Engineer
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <>

View raw message