spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gurwls...@apache.org
Subject spark git commit: [SPARK-22377][BUILD] Use /usr/sbin/lsof if lsof does not exists in release-build.sh
Date Mon, 13 Nov 2017 23:28:48 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.1 ca19271cc -> 7bdad58e2


[SPARK-22377][BUILD] Use /usr/sbin/lsof if lsof does not exists in release-build.sh

## What changes were proposed in this pull request?

This PR proposes to use `/usr/sbin/lsof` if `lsof` is missing in the path to fix nightly snapshot
jenkins jobs. Please refer https://github.com/apache/spark/pull/19359#issuecomment-340139557:

> Looks like some of the snapshot builds are having lsof issues:
>
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.1-maven-snapshots/182/console
>
>https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.2-maven-snapshots/134/console
>
>spark-build/dev/create-release/release-build.sh: line 344: lsof: command not found
>usage: kill [ -s signal | -p ] [ -a ] pid ...
>kill -l [ signal ]

Up to my knowledge,  the full path of `lsof` is required for non-root user in few OSs.

## How was this patch tested?

Manually tested as below:

```bash
#!/usr/bin/env bash

LSOF=lsof
if ! hash $LSOF 2>/dev/null; then
  echo "a"
  LSOF=/usr/sbin/lsof
fi

$LSOF -P | grep "a"
```

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #19695 from HyukjinKwon/SPARK-22377.

(cherry picked from commit c8b7f97b8a58bf4a9f6e3a07dd6e5b0f646d8d99)
Signed-off-by: hyukjinkwon <gurwls223@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7bdad58e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7bdad58e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7bdad58e

Branch: refs/heads/branch-2.1
Commit: 7bdad58e2baac98e7b77f17aaa6c88de230a220e
Parents: ca19271
Author: hyukjinkwon <gurwls223@gmail.com>
Authored: Tue Nov 14 08:28:13 2017 +0900
Committer: hyukjinkwon <gurwls223@gmail.com>
Committed: Tue Nov 14 08:28:43 2017 +0900

----------------------------------------------------------------------
 dev/create-release/release-build.sh | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/7bdad58e/dev/create-release/release-build.sh
----------------------------------------------------------------------
diff --git a/dev/create-release/release-build.sh b/dev/create-release/release-build.sh
index ad32c31..eefd864 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -121,6 +121,13 @@ else
   fi
 fi
 
+# This is a band-aid fix to avoid the failure of Maven nightly snapshot in some Jenkins
+# machines by explicitly calling /usr/sbin/lsof. Please see SPARK-22377 and the discussion
+# in its pull request.
+LSOF=lsof
+if ! hash $LSOF 2>/dev/null; then
+  LSOF=/usr/sbin/lsof
+fi
 
 if [ -z "$SPARK_PACKAGE_VERSION" ]; then
   SPARK_PACKAGE_VERSION="${SPARK_VERSION}-$(date +%Y_%m_%d_%H_%M)-${git_hash}"
@@ -341,7 +348,7 @@ if [[ "$1" == "publish-snapshot" ]]; then
     -DskipTests $PUBLISH_PROFILES clean deploy
 
   # Clean-up Zinc nailgun process
-  lsof -P |grep $ZINC_PORT | grep LISTEN | awk '{ print $2; }' | xargs kill
+  $LSOF -P |grep $ZINC_PORT | grep LISTEN | awk '{ print $2; }' | xargs kill
 
   rm $tmp_settings
   cd ..
@@ -379,7 +386,7 @@ if [[ "$1" == "publish-release" ]]; then
     -DskipTests $PUBLISH_PROFILES clean install
 
   # Clean-up Zinc nailgun process
-  lsof -P |grep $ZINC_PORT | grep LISTEN | awk '{ print $2; }' | xargs kill
+  $LSOF -P |grep $ZINC_PORT | grep LISTEN | awk '{ print $2; }' | xargs kill
 
   ./dev/change-version-to-2.10.sh
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message