spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gurwls...@apache.org
Subject spark git commit: [SPARK-22302][INFRA] Remove manual backports for subprocess and print explicit message for < Python 2.7
Date Sat, 21 Oct 2017 17:22:38 GMT
Repository: spark
Updated Branches:
  refs/heads/master a763607e4 -> ff8de99a1


[SPARK-22302][INFRA] Remove manual backports for subprocess and print explicit message for
< Python 2.7

## What changes were proposed in this pull request?

Seems there was a mistake - missing import for `subprocess.call`, while refactoring this script
a long ago, which should be used for backports of some missing functions in `subprocess`,
specifically in < Python 2.7.

Reproduction is:

```
cd dev && python2.6
```

```
>>> from sparktestsupport import shellutils
>>> shellutils.subprocess_check_call("ls")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sparktestsupport/shellutils.py", line 46, in subprocess_check_call
    retcode = call(*popenargs, **kwargs)
NameError: global name 'call' is not defined
```

For Jenkins logs, please see https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3950/console

Since we dropped the Python 2.6.x support, looks better we remove those workarounds and print
out explicit error messages in order to reduce the efforts to find out the root causes for
such cases, for example, `https://github.com/apache/spark/pull/19513#issuecomment-337406734`.

## How was this patch tested?

Manually tested:

```
./dev/run-tests
```

```
Python versions prior to 2.7 are not supported.
```

```
./dev/run-tests-jenkins
```

```
Python versions prior to 2.7 are not supported.
```

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #19524 from HyukjinKwon/SPARK-22302.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ff8de99a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ff8de99a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ff8de99a

Branch: refs/heads/master
Commit: ff8de99a1c7b4a291e661cd0ad12748f4321e43d
Parents: a763607
Author: hyukjinkwon <gurwls223@gmail.com>
Authored: Sun Oct 22 02:22:35 2017 +0900
Committer: hyukjinkwon <gurwls223@gmail.com>
Committed: Sun Oct 22 02:22:35 2017 +0900

----------------------------------------------------------------------
 dev/run-tests                      |  6 ++++++
 dev/run-tests-jenkins              |  7 ++++++-
 dev/sparktestsupport/shellutils.py | 31 ++-----------------------------
 3 files changed, 14 insertions(+), 30 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/ff8de99a/dev/run-tests
----------------------------------------------------------------------
diff --git a/dev/run-tests b/dev/run-tests
index 257d1e8..9cf93d0 100755
--- a/dev/run-tests
+++ b/dev/run-tests
@@ -20,4 +20,10 @@
 FWDIR="$(cd "`dirname $0`"/..; pwd)"
 cd "$FWDIR"
 
+PYTHON_VERSION_CHECK=$(python -c 'import sys; print(sys.version_info < (2, 7, 0))')
+if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
+  echo "Python versions prior to 2.7 are not supported."
+  exit -1
+fi
+
 exec python -u ./dev/run-tests.py "$@"

http://git-wip-us.apache.org/repos/asf/spark/blob/ff8de99a/dev/run-tests-jenkins
----------------------------------------------------------------------
diff --git a/dev/run-tests-jenkins b/dev/run-tests-jenkins
index f41f1ac..03fd6ff 100755
--- a/dev/run-tests-jenkins
+++ b/dev/run-tests-jenkins
@@ -25,5 +25,10 @@
 FWDIR="$( cd "$( dirname "$0" )/.." && pwd )"
 cd "$FWDIR"
 
-export PATH=/home/anaconda/bin:$PATH
+PYTHON_VERSION_CHECK=$(python -c 'import sys; print(sys.version_info < (2, 7, 0))')
+if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
+  echo "Python versions prior to 2.7 are not supported."
+  exit -1
+fi
+
 exec python -u ./dev/run-tests-jenkins.py "$@"

http://git-wip-us.apache.org/repos/asf/spark/blob/ff8de99a/dev/sparktestsupport/shellutils.py
----------------------------------------------------------------------
diff --git a/dev/sparktestsupport/shellutils.py b/dev/sparktestsupport/shellutils.py
index 05af871..c7644da 100644
--- a/dev/sparktestsupport/shellutils.py
+++ b/dev/sparktestsupport/shellutils.py
@@ -21,35 +21,8 @@ import shutil
 import subprocess
 import sys
 
-
-if sys.version_info >= (2, 7):
-    subprocess_check_output = subprocess.check_output
-    subprocess_check_call = subprocess.check_call
-else:
-    # SPARK-8763
-    # backported from subprocess module in Python 2.7
-    def subprocess_check_output(*popenargs, **kwargs):
-        if 'stdout' in kwargs:
-            raise ValueError('stdout argument not allowed, it will be overridden.')
-        process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
-        output, unused_err = process.communicate()
-        retcode = process.poll()
-        if retcode:
-            cmd = kwargs.get("args")
-            if cmd is None:
-                cmd = popenargs[0]
-            raise subprocess.CalledProcessError(retcode, cmd, output=output)
-        return output
-
-    # backported from subprocess module in Python 2.7
-    def subprocess_check_call(*popenargs, **kwargs):
-        retcode = call(*popenargs, **kwargs)
-        if retcode:
-            cmd = kwargs.get("args")
-            if cmd is None:
-                cmd = popenargs[0]
-            raise CalledProcessError(retcode, cmd)
-        return 0
+subprocess_check_output = subprocess.check_output
+subprocess_check_call = subprocess.check_call
 
 
 def exit_from_command_with_retcode(cmd, retcode):


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message