beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmet Altay (JIRA)" <>
Subject [jira] [Commented] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt
Date Fri, 27 Oct 2017 16:36:01 GMT


Ahmet Altay commented on BEAM-3106:

The reason behind pinning or capping dependencies is to prevent broken releases after time
passes. In the past there were many occasions of a dependency releasing a backward-incompatible
change and breaking and already release Beam version. This is bad for existing Beam users,
because at some point in the future their currently working Beam release may stop and force
them to do upgrades.

We follow the semantic versioning rules (capping) in general with dependencies. (For example:
'avro>=1.8.1,<2.0.0'). However, in the past, some dependencies, also released breaking
changes without incrementing the major version. For those dependencies only, we pin their
version to prevent any future breakages.

We can consider an alternative policy to what we are doing today, but it is important for
us to ensure (as much as possible) that already released Beam versions will continue to work
even after breaking change in a dependency. 

> Consider not pinning all python dependencies, or moving them to requirements.txt
> --------------------------------------------------------------------------------
>                 Key: BEAM-3106
>                 URL:
>             Project: Beam
>          Issue Type: Wish
>          Components: build-system
>    Affects Versions: 2.1.0
>         Environment: python
>            Reporter: Maximilian Roos
>            Assignee: Ahmet Altay
> Currently all python dependencies are [pinned or capped|]
> While there's a good argument for supplying a `requirements.txt` with well tested dependencies,
having them specified in `` forces them to an exact state on each install of Beam.
This makes using Beam in any environment with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have libraries that
won't work with an older version (but Beam _does_ work with an newer version). We have to
do a bunch of gymnastics to get the correct versions installed because of this. Unfortunately,
airflow repeats this practice and conflicts on a number of dependencies, adding further complication
(but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for example
no libraries in numerical python do this. Here's a [discussion on SO|]

This message was sent by Atlassian JIRA

View raw message