Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Fri, 27 Oct 2017 17:14:00 +0000 (UTC)
From: "Maximilian Roos (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13112399.1509050857000.81736.1509124440212@Atlassian.JIRA>
In-Reply-To: <JIRA.13112399.1509050857000@Atlassian.JIRA>
References: <JIRA.13112399.1509050857000@Atlassian.JIRA> <JIRA.13112399.1509050857713@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-3106) Consider not pinning all python
 dependencies, or moving them to requirements.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 27 Oct 2017 17:14:04 -0000


    [ https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222698#comment-16222698 ] 

Maximilian Roos commented on BEAM-3106:
---------------------------------------

Yes, those are good points, and this is a tough problem; particularly for Beam (or airflow) which relies on a lot of relatively fast-moving dependencies. 

That said, the consensus in the python community seems to be that `requirements.txt` is the place to put those pins, rather than `setup.py`, and that putting them in `setup.py` creates more interference than clarity.

Pinning to major versions could be a reasonable compromise. Though as an example, `google-cloud-bigquery` is pinned to a version 3 behind the latest, while it works (or at least I haven't had any issues) with the latest.

Thanks Ahmet.


> Consider not pinning all python dependencies, or moving them to requirements.txt
> --------------------------------------------------------------------------------
>
>                 Key: BEAM-3106
>                 URL: https://issues.apache.org/jira/browse/BEAM-3106
>             Project: Beam
>          Issue Type: Wish
>          Components: build-system
>    Affects Versions: 2.1.0
>         Environment: python
>            Reporter: Maximilian Roos
>            Assignee: Ahmet Altay
>
> Currently all python dependencies are [pinned or capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well tested dependencies, having them specified in `setup.py` forces them to an exact state on each install of Beam. This makes using Beam in any environment with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have libraries that won't work with an older version (but Beam _does_ work with an newer version). We have to do a bunch of gymnastics to get the correct versions installed because of this. Unfortunately, airflow repeats this practice and conflicts on a number of dependencies, adding further complication (but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for example no libraries in numerical python do this. Here's a [discussion on SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)