beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmet Altay (JIRA)" <>
Subject [jira] [Commented] (BEAM-680) Python Dataflow stages stale requirements-cache dependencies
Date Tue, 04 Apr 2017 17:32:41 GMT


Ahmet Altay commented on BEAM-680:

The solution to this is to use `requirements_cache` flag which can be pointed to a clean directory.

Without using that flag there is no clear solution:
- Cleaning up the cache would result in re-downloading same requirements
- pip does not have a way of telling the transitive dependencies (short of what we are already
- Also uploading more file to staging is generally harmless (other than bloat) but the opposite
(e.g missing a dependency) would cause issues at execution time.

> Python Dataflow stages stale requirements-cache dependencies
> ------------------------------------------------------------
>                 Key: BEAM-680
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Scott Wegner
>            Priority: Minor
> When executing a python pipeline using a requirements.txt file, the Dataflow runner will
stage all dependencies downloaded to its requirements cache directory, including those specified
in the requirements.txt, and any previously cached dependencies. This results in bloated staging
directory if previous pipeline runs from the same machine included different dependencies.
> Repro:
> # Initialize a virtualenv and pip install apache_beam
> # Create an empty requirements.txt file
> # Create a simple pipeline using DataflowPipelineRunner and a requirements.txt file,
for example: [|]
> # {{touch /tmp/dataflow-requirements-cache/extra-file.txt}}
> # Run the pipeline with a specified staging directory
> # Check the staged files for the job
> 'extra-file.txt' will be uploaded with the job, along with any other cached dependencies
under /tmp/dataflow-requirements-cache.
> We should only be staging the dependencies necessary for a pipeline, not all previously-cached
dependencies found on the machine.

This message was sent by Atlassian JIRA

View raw message