Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.incubator.apache.org
Date: Mon, 26 Sep 2016 20:28:20 +0000 (UTC)
From: "Scott Wegner (JIRA)" <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.13007694.1474921687000.666858.1474921700481@Atlassian.JIRA>
In-Reply-To: <JIRA.13007694.1474921687000@Atlassian.JIRA>
References: <JIRA.13007694.1474921687000@Atlassian.JIRA> <JIRA.13007694.1474921687377@arcas>
Subject: [jira] [Created] (BEAM-680) Python Dataflow stages stale
 requirements-cache dependencies
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 26 Sep 2016 20:28:26 -0000

Scott Wegner created BEAM-680:
---------------------------------

             Summary: Python Dataflow stages stale requirements-cache dependencies
                 Key: BEAM-680
                 URL: https://issues.apache.org/jira/browse/BEAM-680
             Project: Beam
          Issue Type: Bug
          Components: sdk-py
            Reporter: Scott Wegner
            Priority: Minor


When executing a python pipeline using a requirements.txt file, the Dataflow runner will stage all dependencies downloaded to its requirements cache directory, including those specified in the requirements.txt, and any previously cached dependencies. This results in bloated staging directory if previous pipeline runs from the same machine included different dependencies.

Repro:

# Initialize a virtualenv and pip install apache_beam
# Create an empty requirements.txt file
# Create a simple pipeline using DataflowPipelineRunner and a requirements.txt file, for example: [my_pipeline.py|https://gist.github.com/swegner/6df00df1423b48206c4ab5a7e917218a]
# {{touch /tmp/dataflow-requirements-cache/extra-file.txt}}
# Run the pipeline with a specified staging directory
# Check the staged files for the job

'extra-file.txt' will be uploaded with the job, along with any other cached dependencies under /tmp/dataflow-requirements-cache.

We should only be staging the dependencies necessary for a pipeline, not all previously-cached dependencies found on the machine.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)