beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maximilian Roos (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt
Date Sat, 28 Oct 2017 19:41:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223716#comment-16223716
] 

Maximilian Roos edited comment on BEAM-3106 at 10/28/17 7:40 PM:
-----------------------------------------------------------------

> What do you think about a policy like, reviewing capped dependencies at every release
and ensuring that a) we are including latest versions of these known dependencies, b) we are
testing with those dependencies before a release.

I think that's a reasonable compromise, thanks Ahmet. 

To close this off, here's the system that works really well in the numerical python ecosystem
(e.g. pandas / numpy / xarray):
- Run CI tests on a number of released versions of each dependency
- Any backward incompatible changes in dependencies are deprecated in advance, and tests catch
those deprecation warnings - giving plenty of time for changes (this relies on all dependencies
raising warnings for backward-incompatible changes)
- In the extensively maintained libraries, tests are also run on master branch of dependencies,
to quickly flag any potential breakages




was (Author: maximilianr):

??What do you think about a policy like, reviewing capped dependencies at every release and
ensuring that a) we are including latest versions of these known dependencies, b) we are testing
with those dependencies before a release.
??

I think that's a reasonable compromise, thanks Ahmet. 

To close this off, here's the system that works really well in the numerical python ecosystem
(e.g. pandas / numpy / xarray):
- Run CI tests on a number of released versions of each dependency
- Any backward incompatible changes in dependencies are deprecated in advance, and tests catch
those deprecation warnings - giving plenty of time for changes (this relies on all dependencies
raising warnings for backward-incompatible changes)
- In the extensively maintained libraries, tests are also run on master branch of dependencies,
to quickly flag any potential breakages



> Consider not pinning all python dependencies, or moving them to requirements.txt
> --------------------------------------------------------------------------------
>
>                 Key: BEAM-3106
>                 URL: https://issues.apache.org/jira/browse/BEAM-3106
>             Project: Beam
>          Issue Type: Wish
>          Components: build-system
>    Affects Versions: 2.1.0
>         Environment: python
>            Reporter: Maximilian Roos
>            Assignee: Ahmet Altay
>
> Currently all python dependencies are [pinned or capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well tested dependencies,
having them specified in `setup.py` forces them to an exact state on each install of Beam.
This makes using Beam in any environment with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have libraries that
won't work with an older version (but Beam _does_ work with an newer version). We have to
do a bunch of gymnastics to get the correct versions installed because of this. Unfortunately,
airflow repeats this practice and conflicts on a number of dependencies, adding further complication
(but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for example
no libraries in numerical python do this. Here's a [discussion on SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message