aurora-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Sirois (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AURORA-1717) pants fails to run in integration tests (flaky)
Date Thu, 16 Jun 2016 19:45:05 GMT

    [ https://issues.apache.org/jira/browse/AURORA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334529#comment-15334529
] 

John Sirois commented on AURORA-1717:
-------------------------------------

I did confirm robustness to racyness at the most obvious level by deleting my cached pants
{{0.0.80}} and running tests in 2 workspaces in parallel:

Kill cached pants:
{noformat}
rm -rf ~/.cache/pants/setup/bootstrap-Linux-x86_64/0.0.80 ~/.cache/pants/setup/bootstrap-Linux-x86_64/pants.98GeLz
{noformat}

Run 1:
{noformat}
(master) $ ./pants test src/test/python/apache/aurora/executor -- -v
$ ./pants test src/test/python/apache/aurora/executor -- -v
New python executable in /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.U6D1ul/install/bin/python2.7
Also creating executable in /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.U6D1ul/install/bin/python
Installing setuptools, pip, wheel...done.
...
                     src/test/python/apache/aurora/executor/test_http_lifecycle.py::test_http_lifecycle_wraps_start_and_stop
PASSED
                     
                     ===== 38 passed, 2 skipped in 42.40 seconds ======
                     
13:36:33 00:43     [junit]
13:36:33 00:43   [complete]
               SUCCESS
{noformat}

Run 2:
{noformat}
(jsirois/issues/AURORA-1717) $ ./pants test src/test/python/apache/aurora/executor -- -v
New python executable in /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.rveTCx/install/bin/python2.7
Also creating executable in /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.rveTCx/install/bin/python
Installing setuptools, pip, wheel...done.
...
                     src/test/python/apache/aurora/executor/test_http_lifecycle.py::test_http_lifecycle_wraps_start_and_stop
PASSED
                     
                     ===== 38 passed, 2 skipped in 42.37 seconds ======
                     
13:36:33 00:43     [junit]
13:36:33 00:43   [complete]
               SUCCESS
{noformat}

At the end of this, the {{pants.rveTCx}} version of {{0.0.80}} is orphaned, but that's the
only ill-effect of the concurrent pants bootstraps:
{noformat}
$ ls -lrt ~/.cache/pants/setup/bootstrap-Linux-x86_64
total 60
lrwxrwxrwx 1 jsirois jsirois   86 Apr 18 10:20 virtualenv-13.1.0 -> /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.L5d5Hm/virtualenv-13.1.0
drwx------ 3 jsirois jsirois 4096 Apr 18 10:20 pants.L5d5Hm
drwx------ 3 jsirois jsirois 4096 Apr 18 10:20 pants.b3Ue4R
lrwxrwxrwx 1 jsirois jsirois   76 Apr 18 10:20 0.0.82 -> /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.b3Ue4R/install
drwx------ 3 jsirois jsirois 4096 May  1 17:58 pants.V4zNBD
lrwxrwxrwx 1 jsirois jsirois   76 May  1 17:58 1.0.0 -> /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.V4zNBD/install
lrwxrwxrwx 1 jsirois jsirois   76 May 11 09:32 unspecified -> /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.pjfDAt/install
drwx------ 3 jsirois jsirois 4096 May 11 09:32 pants.pjfDAt
lrwxrwxrwx 1 jsirois jsirois   76 Jun  3 13:26 1.1.0-pre3 -> /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.AZoN7e/install
drwx------ 3 jsirois jsirois 4096 Jun  3 13:26 pants.AZoN7e
lrwxrwxrwx 1 jsirois jsirois   76 Jun 14 15:38 1.1.0-pre5 -> /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.mfljwg/install
drwx------ 3 jsirois jsirois 4096 Jun 14 15:38 pants.mfljwg
drwx------ 3 jsirois jsirois 4096 Jun 16 13:35 pants.U6D1ul
lrwxrwxrwx 1 jsirois jsirois   76 Jun 16 13:35 0.0.80 -> /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.U6D1ul/install
drwx------ 3 jsirois jsirois 4096 Jun 16 13:35 pants.rveTCx
{noformat}

> pants fails to run in integration tests (flaky)
> -----------------------------------------------
>
>                 Key: AURORA-1717
>                 URL: https://issues.apache.org/jira/browse/AURORA-1717
>             Project: Aurora
>          Issue Type: Bug
>          Components: Build
>            Reporter: John Sirois
>            Assignee: John Sirois
>
> As noted by [~StephanErb] in [IRC|http://wilderness.apache.org/channels/?f=aurora/2016-06-15#1466009185],
python tests occasionally fail like so under AuroraBot:
> {noformat}
> ...
>       with temporary_dir() as checkpoint_root:
>         te = AuroraExecutor(
> >           runner_provider=make_provider(checkpoint_root),
>             sandbox_provider=DefaultTestSandboxProvider())
> src/test/python/apache/aurora/executor/test_thermos_executor.py:580: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> src/test/python/apache/aurora/executor/test_thermos_executor.py:193: in make_provider
>     pex_location=thermos_runner_path(),
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> build = True
>     def thermos_runner_path(build=True):
>       if not build:
>         return getattr(thermos_runner_path, 'value', None)
>     
>       if not hasattr(thermos_runner_path, 'value'):
>         pex_dir = safe_mkdtemp()
> >       assert subprocess.call(["./pants", "--pants-distdir=%s" % pex_dir, "binary",
>           "src/main/python/apache/thermos/runner:thermos_runner"]) == 0
> E       assert 1 == 0
> E        +  where 1 = <function call at 0x7f39ef905938>(['./pants', '--pants-distdir=/tmp/user/2396/tmpU8sLiq',
'binary', 'src/main/python/apache/thermos/runner:thermos_runner'])
> E        +    where <function call at 0x7f39ef905938> = subprocess.call
> src/test/python/apache/aurora/executor/test_thermos_executor.py:185: AssertionError
> -------------- Captured stderr call --------------
> Traceback (most recent call last):
> File "/home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/0.0.80/bin/pants", line
7, in <module>
>  from pants.bin.pants_exe import main
> ImportError: No module named pants.bin.pants_exe
> generated xml file: /home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/415337499eb72578eab327a6487c1f5c9452b3d6.xml

>  16 failed, 638 passed, 6 skipped, 1 warnings, 8 error in 127.24 seconds 
> ...
> {noformat}
> This has all the signs of a race in pants bootstrapping; ie: the {{\[version symlink]/bin/pants}}
script is installed but the interpreter it points to in its shebang does not have pants fully
installed yet.  That said, a review of the pants bootstrap script seems to indicate correct
atomicity.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message