spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JoshRosen <>
Subject [GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Backport dev/test...
Date Sun, 10 Jan 2016 06:28:12 GMT
GitHub user JoshRosen opened a pull request:

    [SPARK-10359][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.5

    This patch backports the `dev/test-dependencies` script (from #10461) to branch-1.6.

You can merge this pull request into a Git repository by running:

    $ git pull test-deps-16-backport

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #10680
commit f97d4d56fd7c8c0d0dd417921ed47b7dd1814531
Author: Josh Rosen <>
Date:   2015-12-30T20:47:42Z

    [SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests
    This patch adds a new build check which enumerates Spark's resolved runtime classpath
and saves it to a file, then diffs against that file to detect whether pull requests have
introduced dependency changes. The aim of this check is to make it simpler to reason about
whether pull request which modify the build have introduced new dependencies or changed transitive
dependencies in a way that affects the final classpath.
    This supplants the checks added in SPARK-4123 / #5093, which are currently disabled due
to bugs.
    This patch is based on pwendell's work in #8531.
    Closes #8531.
    Author: Josh Rosen <>
    Author: Patrick Wendell <>
    Closes #10461 from JoshRosen/SPARK-10359.

commit e1ab0440e66431ea4046b408b3b0a3087e0bd78c
Author: Josh Rosen <>
Date:   2016-01-01T04:23:19Z

    [SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/ script
    This patch includes multiple fixes for the `dev/` script (which was
introduced in #10461):
    - Use `build/mvn --force` instead of `mvn` in one additional place.
    - Explicitly set a zero exit code on success.
    - Set `LC_ALL=C` to make `sort` results agree across machines (see
    - Set `should_run_build_tests=True` for `build` module (this somehow got lost).
    Author: Josh Rosen <>
    Closes #10543 from JoshRosen/dep-script-fixes.

commit f8b4fdb91870e0702bd2735f9a5a58dea4a63fff
Author: Josh Rosen <>
Date:   2016-01-04T09:04:29Z

    [SPARK-10359][PROJECT-INFRA] Use more random number in dev/; fix version
    This patch aims to fix another potential source of flakiness in the `dev/`
    pwendell's original patch and my version used `$(date +%s | tail -c6)` to generate a suffix
to use when installing temporary Spark versions into the local Maven cache, but this value
only changes once per second and thus is highly collision-prone when concurrent builds launch
on AMPLab Jenkins. In order to reduce the potential for conflicts, this patch updates the
script to call Python's random number generator instead.
    I also fixed a bug in how we captured the original project version; the bug was causing
the exit handler code to fail.
    Author: Josh Rosen <>
    Closes #10558 from JoshRosen/build-dep-tests-round-3.


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message