beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-5683) [beam_PostCommit_Py_VR_Dataflow] [test_multiple_empty_outputs] Fails due to pip download flake
Date Fri, 12 Oct 2018 18:49:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-5683?focusedWorklogId=153987&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153987
]

ASF GitHub Bot logged work on BEAM-5683:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 12/Oct/18 18:48
            Start Date: 12/Oct/18 18:48
    Worklog Time Spent: 10m 
      Work Description: pabloem closed pull request #6646: [BEAM-5683] Print command logs
on failure
URL: https://github.com/apache/beam/pull/6646
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/portability/stager.py b/sdks/python/apache_beam/runners/portability/stager.py
index ef7401ac6aa..885d69c7866 100644
--- a/sdks/python/apache_beam/runners/portability/stager.py
+++ b/sdks/python/apache_beam/runners/portability/stager.py
@@ -408,7 +408,7 @@ def _populate_requirements_cache(requirements_file, cache_dir):
         ':all:'
     ]
     logging.info('Executing command: %s', cmd_args)
-    processes.check_call(cmd_args)
+    processes.check_output(cmd_args)
 
   @staticmethod
   def _build_setup_package(setup_file, temp_dir, build_setup_args=None):
@@ -421,7 +421,7 @@ def _build_setup_package(setup_file, temp_dir, build_setup_args=None):
             os.path.basename(setup_file), 'sdist', '--dist-dir', temp_dir
         ]
       logging.info('Executing command: %s', build_setup_args)
-      processes.check_call(build_setup_args)
+      processes.check_output(build_setup_args)
       output_files = glob.glob(os.path.join(temp_dir, '*.tar.gz'))
       if not output_files:
         raise RuntimeError(
@@ -549,7 +549,7 @@ def _download_pypi_sdk_package(temp_dir,
 
     logging.info('Executing command: %s', cmd_args)
     try:
-      processes.check_call(cmd_args)
+      processes.check_output(cmd_args)
     except subprocess.CalledProcessError as e:
       raise RuntimeError(repr(e))
 
diff --git a/sdks/python/apache_beam/runners/portability/stager_test.py b/sdks/python/apache_beam/runners/portability/stager_test.py
index 9edc4eb4a2d..43b24967d82 100644
--- a/sdks/python/apache_beam/runners/portability/stager_test.py
+++ b/sdks/python/apache_beam/runners/portability/stager_test.py
@@ -65,7 +65,7 @@ def populate_requirements_cache(self, requirements_file, cache_dir):
     self.create_temp_file(os.path.join(cache_dir, 'def.txt'), 'nothing')
 
   def build_fake_pip_download_command_handler(self, has_wheels):
-    """A stub for apache_beam.utils.processes.check_call that imitates pip.
+    """A stub for apache_beam.utils.processes.check_output that imitates pip.
 
       Args:
         has_wheels: Whether pip fake should have a whl distribution of packages.
@@ -291,7 +291,7 @@ def test_sdk_location_default(self):
     options.view_as(SetupOptions).sdk_location = 'default'
 
     with mock.patch(
-        'apache_beam.utils.processes.check_call',
+        'apache_beam.utils.processes.check_output',
         self.build_fake_pip_download_command_handler(has_wheels=False)):
       _, staged_resources = self.stager.stage_job_resources(
           options, temp_dir=self.make_temp_dir(), staging_location=staging_dir)
@@ -309,7 +309,7 @@ def test_sdk_location_default_with_wheels(self):
     options.view_as(SetupOptions).sdk_location = 'default'
 
     with mock.patch(
-        'apache_beam.utils.processes.check_call',
+        'apache_beam.utils.processes.check_output',
         self.build_fake_pip_download_command_handler(has_wheels=True)):
       _, staged_resources = self.stager.stage_job_resources(
           options, temp_dir=self.make_temp_dir(), staging_location=staging_dir)


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 153987)
    Time Spent: 1h  (was: 50m)

> [beam_PostCommit_Py_VR_Dataflow] [test_multiple_empty_outputs] Fails due to pip download
flake
> ----------------------------------------------------------------------------------------------
>
>                 Key: BEAM-5683
>                 URL: https://issues.apache.org/jira/browse/BEAM-5683
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-harness, test-failures
>            Reporter: Scott Wegner
>            Assignee: Ankur Goenka
>            Priority: Major
>              Labels: currently-failing
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins Job|https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/1289/]
>  * [Gradle Build Scan|https://scans.gradle.com/s/hjmzvh4ylhs6y/console-log?task=:beam-sdks-python:validatesRunnerBatchTests]
>  * [Test source code|https://github.com/apache/beam/blob/303a4275eb0a323761e1a4dec6a22fde9863acf8/sdks/python/apache_beam/runners/portability/stager.py#L390]
> Initial investigation:
> Seems to be failing on pip download.
> ======================================================================
> ERROR: test_multiple_empty_outputs (apache_beam.transforms.ptransform_test.PTransformTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/transforms/ptransform_test.py",
line 277, in test_multiple_empty_outputs
>     pipeline.run()
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/testing/test_pipeline.py",
line 104, in run
>     result = super(TestPipeline, self).run(test_runner_api)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/pipeline.py",
line 403, in run
>     self.to_runner_api(), self.runner, self._options).run(False)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/pipeline.py",
line 416, in run
>     return self.runner.run_pipeline(self)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
line 50, in run_pipeline
>     self.result = super(TestDataflowRunner, self).run_pipeline(pipeline)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
line 389, in run_pipeline
>     self.dataflow_client.create_job(self.job), self)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/utils/retry.py",
line 184, in wrapper
>     return fun(*args, **kwargs)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
line 490, in create_job
>     self.create_job_description(job)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
line 519, in create_job_description
>     resources = self._stage_resour
> ces(job.options)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
line 452, in _stage_resources
>     staging_location=google_cloud_options.staging_location)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/portability/stager.py",
line 161, in stage_job_resources
>     requirements_cache_path)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/portability/stager.py",
line 411, in _populate_requirements_cache
>     processes.check_call(cmd_args)
>   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/utils/processes.py",
line 46, in check_call
>     return subprocess.check_call(*args, **kwargs)
>   File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
>     raise CalledProcessError(retcode, cmd)
> CalledProcessError: Command '['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/build/gradleenv/bin/python',
'-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', 'postcommit_requirements.txt',
'--exists-action', 'i', '--no-binary', ':all:']' returned non-zero exit status 1
> ----
> _After you've filled out the above details, please [assign the issue to an individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
Assignee should [treat test failures as high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
helping to fix the issue or find a more appropriate owner. See [Apache Beam Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message