kudu-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a...@apache.org
Subject kudu git commit: build: allow overriding of pip installation parameters
Date Mon, 06 Aug 2018 18:04:01 GMT
Repository: kudu
Updated Branches:
  refs/heads/master 07c9962cb -> c93f08393


build: allow overriding of pip installation parameters

This commit introduces the PIP_INSTALL_FLAGS environment variable to
build-and-test.sh which can be used to override various pip installation
behaviors. For example, you can pass "-i <URL>" so that pip uses a different
installation URL to retrieve packages.

For this to work effectively, PIP_INSTALL_FLAGS must come after all of
build-and-test.sh's pip installation flags. That's because if the same flag
is passed multiple times on the command line, the last value "wins". To make
this consistent, I modified MVN_FLAGS and GRADLE_FLAGS to behave the same.

I also copied the pytest-timeout dependency from setup.py to
requirements.txt so that "python setup.py test" won't have to download
anything if it was preceeded with "pip install -r requirements.txt". It is
possible to configure setuptools with a custom index URL instead, but the
various[1] approaches[2] involve modifying files rather than passing
command line arguments.

1. https://setuptools.readthedocs.io/en/latest/easy_install.html#configuration-files
2. https://setuptools.readthedocs.io/en/latest/setuptools.html (search for dependency_links)

Change-Id: Ib09da4bb2e2cdcbf7ea4a8ab2c1089ab46d934fb
Reviewed-on: http://gerrit.cloudera.org:8080/11116
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
Reviewed-by: Andrew Wong <awong@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/c93f0839
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/c93f0839
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/c93f0839

Branch: refs/heads/master
Commit: c93f08393d6232d9e8472cb50188bcc88409a510
Parents: 07c9962
Author: Adar Dembo <adar@cloudera.com>
Authored: Fri Aug 3 13:40:44 2018 -0700
Committer: Adar Dembo <adar@cloudera.com>
Committed: Mon Aug 6 18:03:29 2018 +0000

----------------------------------------------------------------------
 build-support/jenkins/build-and-test.sh | 55 ++++++++++++++++------------
 python/requirements.txt                 |  5 +++
 2 files changed, 37 insertions(+), 23 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/c93f0839/build-support/jenkins/build-and-test.sh
----------------------------------------------------------------------
diff --git a/build-support/jenkins/build-and-test.sh b/build-support/jenkins/build-and-test.sh
index 3c64f3a..8eb0cdf 100755
--- a/build-support/jenkins/build-and-test.sh
+++ b/build-support/jenkins/build-and-test.sh
@@ -70,6 +70,10 @@
 #     option is not mutually exclusive from BUILD_PYTHON. If both options
 #     are set (default), then both will be run.
 #
+#   PIP_INSTALL_FLAGS  Default: ""
+#     Extra flags which are passed to 'pip install' when setting up the build
+#     environment for the Python wrapper.
+#
 #   MVN_FLAGS          Default: ""
 #     Extra flags which are passed to 'mvn' when building and running Java
 #     tests. This can be useful, for example, to choose a different maven
@@ -377,27 +381,32 @@ if [ "$BUILD_JAVA" == "1" ]; then
 
   # Run the full Maven build.
   if [ "$BUILD_MAVEN" == "1" ]; then
-      MVN_FLAGS="$MVN_FLAGS -B"
-      MVN_FLAGS="$MVN_FLAGS -Dsurefire.rerunFailingTestsCount=3"
-      MVN_FLAGS="$MVN_FLAGS -Dfailsafe.rerunFailingTestsCount=3"
-      MVN_FLAGS="$MVN_FLAGS -Dmaven.javadoc.skip"
-      if ! mvn $MVN_FLAGS clean verify ; then
-        EXIT_STATUS=1
-        FAILURES="$FAILURES"$'Java Maven build/test failed\n'
-      fi
+    EXTRA_MVN_FLAGS="-B"
+    EXTRA_MVN_FLAGS="$EXTRA_MVN_FLAGS -Dsurefire.rerunFailingTestsCount=3"
+    EXTRA_MVN_FLAGS="$EXTRA_MVN_FLAGS -Dfailsafe.rerunFailingTestsCount=3"
+    EXTRA_MVN_FLAGS="$EXTRA_MVN_FLAGS -Dmaven.javadoc.skip"
+    EXTRA_MVN_FLAGS="$EXTRA_MVN_FLAGS $MVN_FLAGS"
+    if ! mvn $EXTRA_MVN_FLAGS clean verify ; then
+      EXIT_STATUS=1
+      FAILURES="$FAILURES"$'Java Maven build/test failed\n'
+    fi
   fi
 
   # Run the full Gradle build.
   if [ "$BUILD_GRADLE" == "1" ]; then
-     GRADLE_FLAGS="$GRADLE_FLAGS --console=plain --no-daemon --continue -DrerunFailingTestsCount=3"
-     # KUDU-2524: temporarily disable scalafmt until we can work out its JDK
-     # incompatibility issue.
-     GRADLE_FLAGS="$GRADLE_FLAGS -DskipFormat"
-     # TODO: Run `gradle check` in BUILD_TYPE DEBUG when static code analysis is fixed
-     if ! ./gradlew $GRADLE_FLAGS clean test integrationTest ; then
-       EXIT_STATUS=1
-       FAILURES="$FAILURES"$'Java Gradle build/test failed\n'
-     fi
+    EXTRA_GRADLE_FLAGS="--console=plain"
+    EXTRA_GRADLE_FLAGS="$EXTRA_GRADLE_FLAGS --no-daemon"
+    EXTRA_GRADLE_FLAGS="$EXTRA_GRADLE_FLAGS --continue"
+    EXTRA_GRADLE_FLAGS="$EXTRA_GRADLE_FLAGS -DrerunFailingTestsCount=3"
+    # KUDU-2524: temporarily disable scalafmt until we can work out its JDK
+    # incompatibility issue.
+    EXTRA_GRADLE_FLAGS="$EXTRA_GRADLE_FLAGS -DskipFormat"
+    EXTRA_GRADLE_FLAGS="$EXTRA_GRADLE_FLAGS $GRADLE_FLAGS"
+    # TODO: Run `gradle check` in BUILD_TYPE DEBUG when static code analysis is fixed
+    if ! ./gradlew $EXTRA_GRADLE_FLAGS clean test integrationTest ; then
+      EXIT_STATUS=1
+      FAILURES="$FAILURES"$'Java Gradle build/test failed\n'
+    fi
   fi
 
   # Run a script to verify the contents of the JARs to ensure the shading and
@@ -437,20 +446,20 @@ if [ "$BUILD_PYTHON" == "1" ]; then
   # Beginning with pip 10, Python 2.6 is no longer supported. Attempting to
   # upgrade to pip 10 on Python 2.6 yields syntax errors. We don't need any new
   # pip features, so let's pin to the last pip version to support Python 2.6.
-  pip install -i https://pypi.python.org/simple --upgrade 'pip < 10.0.0b1'
+  pip install -i https://pypi.python.org/simple $PIP_INSTALL_FLAGS --upgrade 'pip < 10.0.0b1'
 
   # New versions of pip raise an exception when upgrading old versions of
   # setuptools (such as the one found in el6). The workaround is to upgrade
   # setuptools on its own, outside of requirements.txt, and with the pip version
   # check disabled.
-  pip install --disable-pip-version-check --upgrade 'setuptools >= 0.8'
+  pip install --disable-pip-version-check $PIP_INSTALL_FLAGS --upgrade 'setuptools >=
0.8'
 
   # We've got a new pip and new setuptools. We can now install the rest of the
   # Python client's requirements.
   #
   # Installing the Cython dependency may involve some compiler work, so we must
   # pass in the current values of CC and CXX.
-  CC=$CLANG CXX=$CLANG++ pip install -r requirements.txt
+  CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS -r requirements.txt
 
   # Delete old Cython extensions to force them to be rebuilt.
   rm -Rf build kudu_python.egg-info kudu/*.so
@@ -495,20 +504,20 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
   # recursively to transitive dependencies installed via a direct dependency's
   # "python setup.py" command. Therefore we have no choice but to upgrade to a
   # new version of pip to proceed.
-  pip install -i https://pypi.python.org/simple --upgrade pip
+  pip install -i https://pypi.python.org/simple $PIP_INSTALL_FLAGS --upgrade pip
 
   # New versions of pip raise an exception when upgrading old versions of
   # setuptools (such as the one found in el6). The workaround is to upgrade
   # setuptools on its own, outside of requirements.txt, and with the pip version
   # check disabled.
-  pip install --disable-pip-version-check --upgrade 'setuptools >= 0.8'
+  pip install --disable-pip-version-check $PIP_INSTALL_FLAGS --upgrade 'setuptools >=
0.8'
 
   # We've got a new pip and new setuptools. We can now install the rest of the
   # Python client's requirements.
   #
   # Installing the Cython dependency may involve some compiler work, so we must
   # pass in the current values of CC and CXX.
-  CC=$CLANG CXX=$CLANG++ pip install -r requirements.txt
+  CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS -r requirements.txt
 
   # Delete old Cython extensions to force them to be rebuilt.
   rm -Rf build kudu_python.egg-info kudu/*.so

http://git-wip-us.apache.org/repos/asf/kudu/blob/c93f0839/python/requirements.txt
----------------------------------------------------------------------
diff --git a/python/requirements.txt b/python/requirements.txt
index f7013a9..5b6cbc1 100644
--- a/python/requirements.txt
+++ b/python/requirements.txt
@@ -3,6 +3,11 @@
 # See https://docs.pytest.org/en/latest/changelog.html#id164 for more details.
 pytest >=2.8,<3.3
 
+# pytest-timeout 1.2.1 dropped support for python 2.6.
+#
+# See https://pypi.org/project/pytest-timeout/#id5 for more details.
+pytest-timeout >=1.1.0,<1.2.1
+
 cython ==0.26.1
 six
 unittest2


Mime
View raw message