yetus-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a.@apache.org
Subject [yetus] 01/02: YETUS-723. Overhaul the Docker support for 2018
Date Sun, 16 Dec 2018 20:10:07 GMT
This is an automated email from the ASF dual-hosted git repository.

aw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yetus.git

commit 0f7682be3e766f5fa6c459413f7b8a08eb243fce
Author: Allen Wittenauer <aw@apache.org>
AuthorDate: Wed Nov 14 10:17:41 2018 -0800

    YETUS-723. Overhaul the Docker support for 2018
    
    Signed-off-by: Allen Wittenauer <aw@apache.org>
---
 .circleci/config.yml                               |   2 +
 Dockerfile                                         |   1 +
 .../source/documentation/in-progress.html.md       |   1 +
 .../in-progress/precommit-advanced.md              |  27 --
 .../documentation/in-progress/precommit-basic.md   |   2 +-
 .../documentation/in-progress/precommit-docker.md  |  76 ++++
 precommit/src/main/shell/core.d/00-yetuslib.sh     |  59 +++
 precommit/src/main/shell/core.d/01-common.sh       |  14 +-
 .../src/main/shell/core.d/builtin-bugsystem.sh     |   1 +
 precommit/src/main/shell/core.d/docker.sh          | 395 ++++++++++++++-------
 .../src/main/shell/test-patch-docker/Dockerfile    |  17 +-
 ...Dockerfile-endstub => Dockerfile.patchspecific} |  20 +-
 .../shell/test-patch-docker/launch-test-patch.sh   |  69 ++--
 precommit/src/main/shell/test-patch.d/ant.sh       |  10 +-
 .../src/main/shell/test-patch.d/briefreport.sh     |   6 +-
 precommit/src/main/shell/test-patch.d/gradle.sh    |   2 +-
 precommit/src/main/shell/test-patch.d/htmlout.sh   |   3 +-
 precommit/src/main/shell/test-patch.d/maven.sh     |   8 +-
 precommit/src/main/shell/test-patch.sh             | 104 +-----
 19 files changed, 501 insertions(+), 316 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index d225443..a14ce0c 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -26,6 +26,8 @@ jobs:
     environment:
       # Customize the JVM maximum heap limit
       MAVEN_OPTS: -Xmx3200m
+      # Set JAVA_HOME
+      JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
 
     steps:
       - checkout
diff --git a/Dockerfile b/Dockerfile
index 85cd453..ebc2f5e 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -17,6 +17,7 @@ ARG DOCKER_TAG=latest
 ARG DOCKER_REPO=apache/yetus
 FROM ${DOCKER_REPO}-base:${DOCKER_TAG}
 
+LABEL org.apache.yetus=""
 COPY . /ysrc/
 
 # hadolint ignore=DL3003
diff --git a/asf-site-src/source/documentation/in-progress.html.md b/asf-site-src/source/documentation/in-progress.html.md
index 526a2d8..fbfab4b 100644
--- a/asf-site-src/source/documentation/in-progress.html.md
+++ b/asf-site-src/source/documentation/in-progress.html.md
@@ -25,6 +25,7 @@ The Yetus Precommit Patch Tester allows projects to codify their patch acceptanc
 * For an overview of Yetus' philosophy on testing contributions and how evaluation is performed, see our [overview](precommit-architecture).
 * To get started on your project, including an explanation of what we'll expect in a runtime environment and what optional utilities we'll leverage, read through the [basic usage guide](precommit-basic).
 * Customize how precommit interacts with your project by choosing amongst [build systems](precommit-buildtools), [bug systems](precommit-bugsystems) and [test formats](precommit-testformats).
+* Empower the test environment using [Docker](precommit-docker).
 * If your project has advanced requirements such as module relationships not expressed in Maven, special profiles, or a need on os-specific prerequisites not managed by Maven then you'll need to use our [advanced usage guide](precommit-advanced).
 
 For a complete guide to the Precommit API, see [the generated API documentation](precommit-apidocs/).
diff --git a/asf-site-src/source/documentation/in-progress/precommit-advanced.md b/asf-site-src/source/documentation/in-progress/precommit-advanced.md
index a2fe146..98a8fd9 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-advanced.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-advanced.md
@@ -20,38 +20,11 @@
 test-patch
 ==========
 
-* [Docker Support](#docker-support)
 * [Process Reaper](#test-reaper)
 * [Plug-ins](#plug-ins)
 * [Personalities](#personalities)
 * [Important Variables](#important-variables)
 
-# Docker Support
-
-By default, test-patch runs in the same shell where it was launched.  It can alternatively use Docker to launch itself in a container.  This is particularly useful if running under a QA environment that does not provide all the necessary binaries. For example, if the patch requires a newer version of Java than what is installed on a CI instance.
-
-The `--docker` parameter tells test-patch to run in Docker mode. The `--dockerfile` parameter allows one to provide a custom Dockerfile. The Dockerfile should contain all of the necessary binaries and tooling needed to run the test.  test-patch will copy this file up until the text "YETUS CUT HERE" to a different directory and then append its necessary hooks to re-launch itself prior to executing docker.
-
-If a custom Dockerfile cannot be used or the docker executable does not work, test-patch will attempt to recover by switching to its bundled Dockerfile or disabling docker support and running locally.  This behavior can be changed with the `--dockeronfail` option.  It takes a list of comma-delimited settings:
-
-  * fallback - Use the bundled Dockerfile
-  * continue - Turn off docker support
-  * fail - fail the test
-
-The 'fail' setting is always the last option that test-patch will use and may be omitted unless it is the only option.
-
-For example, `--dockeronfail=continue` means if the Dockerfile can't be found, just turn off Docker support and continue running.  `--dockeronfail=fallback` will switch to the bundled Dockerfile and then fail the build if docker fails to work. `--dockeronfail=fail` means to just fail the build and do not try any other mechanisms of recovery. The default is 'fallback,continue,fail' which will allow test-patch to try to continue executing as much as it possibily can.
-
-Be aware that if the Dockerfile is found and the docker command works, test-patch will always fail the build if the Dockerfile itself fails the build.  It will not attempt to continue in the non-Docker mode.
-
-Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in --robot mode.  In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system.  The docker mode can also be run in a "safe" mode that prevents deletions via the `--dockerdelrep` option.   [...]
-
-## Resource Controls
-
-Docker's `--memory` flag is supported via the `--dockermemlimit` option.  This enables the container's memory size to be limited.  This may be important to set to prevent things like broken unit tests bringing down the entire build server.  See [the Docker documentation](https://docs.docker.com/engine/admin/resource_constraints/) for more details. Apache Yetus also sets the --oom-score-adj to 500 in order to offer itself as the first processes to be killed if memory is low.
-
-Additionally, if bash v4 and Linux is in use, a separate process is launched to keep a rolling count of the maximum number of threads (not processes!) in use at one time. This number will be reported at the end of the test-patch run.  Depending upon the build, languages, features enabled, etc, this number may be helpful in determining what the value of `--proclimit`
-
 # Process Reaper
 
 A common problem is the 'stuck' unit test. If bash v4.0 or higher is in use, Apache Yetus may be told to turn on the process reaper functionality.  Using the `--reapearmode` option, this feature may be configured to either report and even kill left over processes that match provided regular expressions.
diff --git a/asf-site-src/source/documentation/in-progress/precommit-basic.md b/asf-site-src/source/documentation/in-progress/precommit-basic.md
index fe3348d..5e24d28 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-basic.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-basic.md
@@ -332,7 +332,7 @@ NOTE: JAVA\_HOME is always appended to the list of JDKs in MultiJDK mode.  If JA
 $ test-patch. (other options) --docker
 ```
 
-This will do some preliminary setup and then re-execute itself inside a Docker container.  For more information on how to provide a custom Dockerfile and other Docker-specific features, see the advanced guide.
+This will do some preliminary setup and then re-execute itself inside a Docker container.  For more information on how to provide a custom Dockerfile and other Docker-specific features, see [precommit-docker](../precommit-docker) for more information.
 
 # In Closing
 
diff --git a/asf-site-src/source/documentation/in-progress/precommit-docker.md b/asf-site-src/source/documentation/in-progress/precommit-docker.md
new file mode 100644
index 0000000..86c4074
--- /dev/null
+++ b/asf-site-src/source/documentation/in-progress/precommit-docker.md
@@ -0,0 +1,76 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+test-patch Docker Support
+=========================
+
+* [The Basics](#the-basics)
+* [Docker Base Images](#docker-base-images)
+  * [Default Images](#default-image)
+  * [Using a Dockerfile](#using-a-dockerfile)
+  * [Pulling a Docker Tag](#pulling-a-docker-tag)
+  * [Platforms](#platforms)
+* [Resource Controls](#resource-controls)
+* [Privileged Mode](#privileged-mode)
+* [Docker-in-Docker](#docker-in-docker)
+
+
+# The Basics
+
+By default, test-patch runs in the same shell where it was launched.  It can alternatively use Docker to launch itself in a Linux container by using the `--docker` parameter.  This is particularly useful if running under a QA environment that does not provide all the necessary binaries. For example, if the patch requires a newer version of Java than what is installed on a CI instance.
+
+Each run will spawn two Docker images, one that contains some sort of base image and one specific to each run.  The base image is described further in this text.  The run-specific image is a small one that passes parameters and settings that are dedicated to that run, with "tp-" as part of the Docker image tag.  It should be removed automatically after the run upon test-patch completion.
+
+# Docker Base Images
+
+## Default Image
+
+By default, test-patch will try to pull apache/yetus:VERSION from the default repository, where VERSION matches the version of Apache Yetus being utilized.  If that fails, it will then build an image based upon the built-in Dockerfile.  Both images contain all of the basic requirements for all of the plug-ins that test-patch supports.  As a result, it is a fairly hefty image!  It may take several minutes to either download or build, dependent upon processor, network speed, etc.
+
+## Using a Dockerfile
+
+The `--dockerfile` parameter allows one to provide a custom Dockerfile instead. The Dockerfile should contain all of the necessary binaries and tooling needed to build and test.  test-patch will process this file up until the text "YETUS CUT HERE".  Be aware that will always fail the build if the Dockerfile itself fails the build.  This makes it ideal to use to test any tools Dockerfile that is also used for development.
+
+Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in `--robot` mode.  In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system.  The docker mode can also be run in a "safe" mode that prevents deletions via the `--dockerdelrep` option. [...]
+
+### COPY and ADD in Dockerfiles
+
+In order to use both 'YETUS CUT HERE' and a Dockerfile that uses COPY and ADD directives, the Docker API must be version 18 or higher.  If the API version is 17 or lower, the Dockerfile will be copied to a temporary directory to be processed, thus removing the Docker build context in the process.
+
+## Pulling a Docker tag
+
+Instead of processing a Dockerfile, test-patch can pull a tag from a repository using the `--docker-tag` parameter. Note that the repository must already be logged into and configured prior to executing test-patch.
+
+## Platforms
+
+When either building or pull an image, test-patch supports the `--docker-platform` flag to pass in the Docker `--platform` flag.  This allows you full control over what kind of image the software either creates or fetches.
+
+# Resource Controls
+
+Docker's `--memory` flag is supported via the `--dockermemlimit` option.  This enables the container's memory size to be limited.  This may be important to set to prevent things like broken unit tests bringing down the entire build server.  See [the Docker documentation](https://docs.docker.com/engine/admin/resource_constraints/) for more details. Apache Yetus also sets the `--oom-score-adj` to 500 in order to offer itself as the first processes to be killed if memory is low.
+
+Additionally, if bash v4 and Linux is in use, a separate process is launched to keep a rolling count of the maximum number of threads (not processes!) in use at one time. This number will be reported at the end of the test-patch run.  Depending upon the build, languages, features enabled, etc, this number may be helpful in determining what the value of `--proclimit`
+
+# Privileged Mode
+
+In some cases, the work being preformed inside the Docker container requires extra permissions.  Using the `--docker-privd` option enables Docker's extended privileges to the container.
+
+# Docker in Docker
+
+With the usage of the `--dockerind` flag, test-patch will mount the `/var/run/docker.sock` UNIX socket into the container to enable Docker-in-Docker mode.  Additionally, the `--docker-socket` option will let one set the socket to mount in situations where that isn't the location of the socket, such as a dockerd proxy providing authentication.
diff --git a/precommit/src/main/shell/core.d/00-yetuslib.sh b/precommit/src/main/shell/core.d/00-yetuslib.sh
index 27d169f..7f21bb3 100755
--- a/precommit/src/main/shell/core.d/00-yetuslib.sh
+++ b/precommit/src/main/shell/core.d/00-yetuslib.sh
@@ -131,6 +131,29 @@ function yetus_abs
   return 1
 }
 
+## @description is a given path relative to given dirpath?
+## @audience    public
+## @stability   stable
+## @replaceable yes
+## @param       dirpath
+## @param       filepath
+## @return      1 - no, path
+## @return      0 - yes, path - dirpath
+function yetus_relative_dir
+{
+  declare dir=$1
+  declare path=$2
+  declare p=${path#${dir}}
+
+  if [[ ${#p} -eq ${#path} ]]; then
+    echo "${p}"
+    return 1
+  fi
+  p=${p#/}
+  echo "${p}"
+  return 0
+}
+
 ## @description  Add a header to the usage output
 ## @audience     public
 ## @stability    evolving
@@ -245,6 +268,42 @@ function yetus_comma_to_array
   IFS="${oldifs}"
 }
 
+## @description  Convert a file to an array.
+## @description  Comments on the beginning of the line are stripped.
+## @audience     public
+## @stability    evolving
+## @replaceable  no
+## @param        arrayname
+## @param        file
+## @return       0 for success
+## @return       1+ for failure
+function yetus_file_to_array
+{
+  declare var=$1
+  declare filename=$2
+  declare line
+  declare a
+
+  if [[ ! -f "${filename}" ]]; then
+    yetus_error "ERROR: ${filename} cannot be read."
+    return 1
+  fi
+
+  if [[ "${BASH_VERSINFO[0]}" -gt 3 ]]; then
+    # Using a pipe to input into mapfile doesn't
+    # work due to the variable only being present in
+    # the subshell.  So MUST force the grep into the
+    # subshell...
+    mapfile -t a < <("${GREP:-grep}" -v -e '^#' "${filename}" )
+  else
+    while read -r line; do
+      a+=("${line}")
+    done < <("${GREP:-grep}" -v -e '^#' "${filename}")
+  fi
+  eval "${var}=(\"\${a[@]}\")"
+  return 0
+}
+
 ## @description  Check if an array has a given value
 ## @audience     public
 ## @stability    stable
diff --git a/precommit/src/main/shell/core.d/01-common.sh b/precommit/src/main/shell/core.d/01-common.sh
index de027e5..e292336 100755
--- a/precommit/src/main/shell/core.d/01-common.sh
+++ b/precommit/src/main/shell/core.d/01-common.sh
@@ -57,6 +57,8 @@ function common_defaults
   #shellcheck disable=SC2034
   PATCH_SYSTEM=""
   PROJECT_NAME=unknown
+  # seed $RANDOM
+  RANDOM=$$
   RESULT=0
   #shellcheck disable=SC2034
   ROBOT=false
@@ -605,14 +607,10 @@ function faster_dirname
 ## @replaceable  no
 function determine_user
 {
-  # On the Apache Jenkins hosts, $USER is pretty much untrustable because
-  # something sets it to an account that doesn't actually exist.
-  # Instead, we need to try and override it with something that's
-  # probably close to reality.
-  if [[ ${TESTPATCHMODE} =~ jenkins ]]; then
-    USER=$(id | cut -f2 -d\( | cut -f1 -d\))
-  fi
-
+  # in some situations, USER isn't properly defined
+  # (e.g., Jenkins).  bash doesn't like you to set this
+  # so don't spit any errors to the screen
+  USER=$(id -u -n) 2>/dev/null
   USER_NAME=${SUDO_USER:=$USER}
   # shellcheck disable=SC2034
   USER_ID=$(id -u "${USER_NAME}")
diff --git a/precommit/src/main/shell/core.d/builtin-bugsystem.sh b/precommit/src/main/shell/core.d/builtin-bugsystem.sh
index a20ccc4..76ed02c 100755
--- a/precommit/src/main/shell/core.d/builtin-bugsystem.sh
+++ b/precommit/src/main/shell/core.d/builtin-bugsystem.sh
@@ -181,5 +181,6 @@ function console_docker_support
 {
   if [[ -n "${CONSOLE_REPORT_FILE}" ]]; then
     DOCKER_EXTRAARGS+=("-v" "${CONSOLE_REPORT_FILE}:/testptch/console.txt")
+    USER_PARAMS+=("--console-report-file=/testptch/console.txt")
   fi
 }
diff --git a/precommit/src/main/shell/core.d/docker.sh b/precommit/src/main/shell/core.d/docker.sh
index 794e610..ea51e3e 100755
--- a/precommit/src/main/shell/core.d/docker.sh
+++ b/precommit/src/main/shell/core.d/docker.sh
@@ -19,18 +19,24 @@ DOCKERCMD=$(command -v docker 2>/dev/null)
 DOCKER_ID=${RANDOM}
 DOCKER_DESTRUCTIVE=true
 DOCKERFILE_DEFAULT="${BINDIR}/test-patch-docker/Dockerfile"
-DOCKERFAIL="fallback,continue,fail"
 DOCKERSUPPORT=false
-DOCKER_ENABLE_PRIVILEGED=true
+DOCKER_ENABLE_PRIVILEGED=false
 DOCKER_CLEANUP_CMD=false
 DOCKER_MEMORY="4g"
+DOCKER_PLATFORM=""
+DOCKER_TAG=""
+DOCKER_IN_DOCKER=false
+DOCKER_SOCKET="/var/run/docker.sock"
 
 declare -a DOCKER_EXTRAARGS
+declare -a DOCKER_VERSION
 
 DOCKER_EXTRAENVS+=("JAVA_HOME")
 DOCKER_EXTRAENVS+=("PATCH_SYSTEM")
 DOCKER_EXTRAENVS+=("PROJECT_NAME")
 
+YETUS_DOCKER_BASH_DEBUG=false
+
 ####
 #### IMPORTANT
 ####
@@ -54,9 +60,13 @@ function docker_usage
   fi
   yetus_add_option "--dockercmd=<file>" "Command to use as docker executable (default: '${DOCKERCMD}')"
   if [[ "${DOCKER_CLEANUP_CMD}" == false ]]; then
+    yetus_add_option "--docker-bash-debug=<bool>" "Enable bash -x mode running in a container (default: ${YETUS_DOCKER_BASH_DEBUG})"
     yetus_add_option "--dockerfile=<file>" "Dockerfile fragment to use as the base (default: '${DOCKERFILE_DEFAULT}')"
-    yetus_add_option "--dockeronfail=<list>" "If Docker fails, determine fallback method order (default: ${DOCKERFAIL})"
+    yetus_add_option "--dockerind=<file>" "Enable Docker-in-Docker by mounting the Docker socket in the container (default: '${DOCKER_IN_DOCKER}')"
+    yetus_add_option "--docker-platform=<plat>" "Use a platform string for building and pulling (default: ${DOCKER_PLATFORM})"
     yetus_add_option "--dockerprivd=<bool>" "Run docker in privileged mode (default: '${DOCKER_ENABLE_PRIVILEGED}')"
+    yetus_add_option "--docker-socket=<socket>" "Mount given socket as /var/run/docker.sock into the container when Docker-in-Docker mode is enabled (default: '${DOCKER_SOCKET}')"
+    yetus_add_option "--docker-tag=<tag>" "Use the given Docker tag as the base"
   fi
   yetus_add_option "--dockerdelrep" "In Docker mode, only report image/container deletions, not act on them"
   if [[ "${DOCKER_CLEANUP_CMD}" == false ]]; then
@@ -78,6 +88,10 @@ function docker_parse_args
       --docker)
         DOCKERSUPPORT=true
       ;;
+      --docker-bash-debug=*)
+        YETUS_DOCKER_BASH_DEBUG=${i#*=}
+        add_docker_env YETUS_DOCKER_BASH_DEBUG
+      ;;
       --dockercmd=*)
         #shellcheck disable=SC2034
         DOCKERCMD=${i#*=}
@@ -88,18 +102,27 @@ function docker_parse_args
       --dockerfile=*)
         DOCKERFILE=${i#*=}
       ;;
+      --dockerind=*)
+        DOCKER_IN_DOCKER=${i#*=}
+      ;;
       --dockermemlimit=*)
         DOCKER_MEMORY=${i#*=}
       ;;
       --dockermode)
         DOCKERMODE=true
       ;;
-      --dockeronfail=*)
-        DOCKERFAIL=${i#*=}
+      --docker-platform=*)
+        DOCKER_PLATFORM=${i#*=}
       ;;
       --dockerprivd=*)
         DOCKER_ENABLE_PRIVILEGED=${i#*=}
       ;;
+      --docker-socket=*)
+        DOCKER_SOCKET=${i#*=}
+      ;;
+      --docker-tag=*)
+        DOCKER_TAG=${i#*=}
+      ;;
     esac
   done
 }
@@ -111,6 +134,7 @@ function docker_parse_args
 function docker_initialize
 {
   declare dockvers
+  declare -a footer
 
   # --docker and --dockermode are mutually
   # exclusive.  --docker is used by the user to
@@ -125,8 +149,18 @@ function docker_initialize
 
   # we are already in docker mode
   if [[ "${DOCKERMODE}" == true ]]; then
-    # DOCKER_VERSION is set by our creator.
-    add_footer_table "Docker" "${DOCKER_VERSION}"
+    # DOCKER_VERSION_STR is set by our creator.
+    footer=("${DOCKER_VERSION_STR}")
+
+    if [[ -n "${DOCKERFROM}" ]]; then
+      footer+=("base:" "${DOCKERFROM}")
+    elif [[ -f "${PATCH_DIR}/Dockerfile" ]]; then
+      footer+=("base:" "@@BASE@@/Dockerfile")
+    fi
+    if [[ -n "${DOCKER_PLATFORM}" ]]; then
+      footer+=("platform:" "${DOCKER_PLATFORM}")
+    fi
+    add_footer_table "Docker" "${footer[*]}"
     return
   fi
 
@@ -135,40 +169,12 @@ function docker_initialize
     return
   fi
 
-  # turn DOCKERFAIL into a string composed of numbers
-  # to ease interpretation:  123, 213, 321, ... whatever
-  # some of these combos are non-sensical but that's ok.
-  # we'll treat non-sense as effectively errors.
-  DOCKERFAIL=${DOCKERFAIL//,/ }
-  DOCKERFAIL=${DOCKERFAIL//fallback/1}
-  DOCKERFAIL=${DOCKERFAIL//continue/2}
-  DOCKERFAIL=${DOCKERFAIL//fail/3}
-  DOCKERFAIL=${DOCKERFAIL//[[:blank:]]/}
-
-  if ! docker_exeverify; then
-    if [[ "${DOCKERFAIL}" =~ ^12
-       || "${DOCKERFAIL}" =~ ^2 ]]; then
-      add_vote_table 0 docker "Docker command '${DOCKERCMD}' not found/broken. Disabling docker."
-      DOCKERSUPPORT=false
-    else
-      add_vote_table -1 docker "Docker command '${DOCKERCMD}' not found/broken."
-      bugsystem_finalreport 1
-      cleanup_and_exit 1
-    fi
-  fi
-
   dockvers=$(docker_version Client)
-  if [[ "${dockvers}" =~ ^0
-     || "${dockvers}" =~ ^1\.[0-5]$ || "${dockvers}" =~ ^1\.[0-5]\. ]]; then
-    if [[ "${DOCKERFAIL}" =~ ^12
-       || "${DOCKERFAIL}" =~ ^2 ]]; then
-      add_vote_table 0 docker "Docker command '${DOCKERCMD}' is too old (${dockvers} < 1.6.0). Disabling docker."
-      DOCKERSUPPORT=false
-    else
-      add_vote_table -1 docker "Docker command '${DOCKERCMD}' is too old (${dockvers} < 1.6.0). Disabling docker."
-      bugsystem_finalreport 1
-      cleanup_and_exit 1
-    fi
+  IFS='.' read -r -a DOCKER_VERSION <<< "${dockvers}"
+  if [[ "${DOCKER_VERSION[0]}" -lt 17 ]]; then
+    add_vote_table -1 docker "Docker command '${DOCKERCMD}' is too old (${dockvers} < 17.0)."
+    bugsystem_finalreport 1
+    cleanup_and_exit 1
   fi
 }
 
@@ -186,24 +192,29 @@ function docker_fileverify
       if [[ -f ${DOCKERFILE} ]]; then
         DOCKERFILE=$(yetus_abs "${DOCKERFILE}")
       else
-        if [[ "${DOCKERFAIL}" =~ ^1 ]]; then
-          yetus_error "ERROR: Dockerfile '${DOCKERFILE}' not found, falling back to built-in."
-          add_vote_table 0 docker "Dockerfile '${DOCKERFILE}' not found, falling back to built-in."
-          DOCKERFILE=${DOCKERFILE_DEFAULT}
-        elif [[ "${DOCKERFAIL}" =~ ^2 ]]; then
-          yetus_error "ERROR: Dockerfile '${DOCKERFILE}' not found, disabling docker."
-          add_vote_table 0 docker "Dockerfile '${DOCKERFILE}' not found, disabling docker."
-          DOCKERSUPPORT=false
-        else
-          yetus_error "ERROR: Dockerfile '${DOCKERFILE}' not found."
-          add_vote_table -1 docker "Dockerfile '${DOCKERFILE}' not found."
-          bugsystem_finalreport 1
-          cleanup_and_exit 1
-        fi
+        yetus_error "ERROR: Dockerfile '${DOCKERFILE}' not found."
+        add_vote_table -1 docker "Dockerfile '${DOCKERFILE}' not found."
+        bugsystem_finalreport 1
+        cleanup_and_exit 1
       fi
       popd >/dev/null || return 1
+    elif [[ -n "${DOCKER_TAG}" ]]; then
+      # do this later as part of the docker handler
+      :
     else
-      DOCKERFILE=${DOCKERFILE_DEFAULT}
+      if [[ -n "${DOCKER_PLATFORM}" ]]; then
+        dockplat=('--platform' "${DOCKER_PLATFORM}")
+      fi
+
+      echo "No --dockerfile or --dockertag provided. Attempting to pull apache/yetus:${VERSION}."
+
+      if dockercmd pull "${dockplat[@]}" "apache/yetus:${VERSION}"; then
+        echo "Pull succeeded; will build with pulled image."
+        DOCKER_TAG="apache/yetus:${VERSION}"
+      else
+        echo "Pull failed; will build with built-in Dockerfile."
+        DOCKERFILE=${DOCKERFILE_DEFAULT}
+      fi
     fi
   fi
 }
@@ -268,12 +279,10 @@ function dockerdate_to_ctime
 
   # believe it or not, date is not even close to standardized...
   if [[ $(uname -s) == Linux ]]; then
-
     # GNU date
     date -d "${mytime}" "+%s"
   else
-
-    # BSD date; docker gives us two different format because fun
+    # BSD date; go/docker gives us two different format because fun
     if ! date -j -f "%FT%T%z" "${mytime}" "+%s" 2>/dev/null; then
       date -j -f "%FT%T" "${mytime}" "+%s"
     fi
@@ -364,7 +373,7 @@ function docker_container_maintenance
 
   done < <(
      # shellcheck disable=SC2086
-     dockercmd inspect \
+     dockercmd container inspect \
         --format '{{.Id}},{{.State.Status}},{{.Created}},{{.State.StartedAt}},{{.State.FinishedAt}},{{.Name}}' \
        ${data})
 }
@@ -390,7 +399,7 @@ function docker_image_maintenance_helper
   fi
 
   for id in "$@"; do
-    tmptime=$(dockercmd inspect --format '{{.Created}}' "${id}" | cut -f1 -d. )
+    tmptime=$(dockercmd image inspect --format '{{.Created}}' "${id}" | cut -f1 -d. )
     createtime=$(dockerdate_to_ctime "${tmptime}")
     curtime=$(date "+%s")
 
@@ -414,8 +423,7 @@ function docker_get_sentinel_images
   dockercmd images \
     | tail -n +2 \
     | "${GREP}" -v hours \
-    | "${AWK}" '{print $1":"$2}' \
-    | "${GREP}" -v "<none>:<none>"
+    | "${AWK}" '{print $3}'
 }
 
 ## @description  Remove untagged/unused images
@@ -442,17 +450,8 @@ function docker_image_maintenance
 
   echo "Apache Yetus images:"
 
-  # removing this by image id doesn't always work without a force
-  # in the situations that, for whatever reason, docker decided
-  # to use the same image. this was a rare problem with older
-  # releases of yetus. at some point, we should revisit this
-  # in the mean time, we're going to reconstruct the
-  # repostory:tag and send that to get removed.
-
-  #shellcheck disable=SC2046,SC2016
-  docker_image_maintenance_helper $(dockercmd images | "${GREP}" -e ^yetus | "${GREP}" tp- | "${AWK}" '{print $1":"$2}')
-  #shellcheck disable=SC2046,SC2016
-  docker_image_maintenance_helper $(dockercmd images | "${GREP}" -e ^yetus | "${GREP}" -v hours | "${AWK}" '{print $1":"$2}')
+  #shellcheck disable=SC2046
+  docker_image_maintenance_helper $(dockercmd images --filter "label=org.apache.yetus" -q --no-trunc)
 
   if [[ "${SENTINEL}" = false ]]; then
     return
@@ -473,24 +472,11 @@ function docker_image_maintenance
 ## @param        args
 function docker_cleanup
 {
-
   docker_image_maintenance
 
   docker_container_maintenance
 }
 
-## @description  Determine the revision of a dockerfile
-## @audience     private
-## @stability    evolving
-## @replaceable  no
-## @param        args
-function docker_getfilerev
-{
-  ${GREP} 'YETUS_PRIVATE: gitrev=' \
-        "${PATCH_DIR}/precommit/test-patch-docker/Dockerfile" \
-          | cut -f2 -d=
-}
-
 ## @description  determine the docker version
 ## @stability    stable
 ## @audience     private
@@ -546,7 +532,7 @@ function docker_do_env_adds
 ## @param        args
 function docker_run_image
 {
-  declare dockerfilerev
+  declare gitfilerev
   declare baseimagename
   declare patchimagename="yetus/${PROJECT_NAME}:tp-${DOCKER_ID}"
   declare containername="yetus_tp-${DOCKER_ID}"
@@ -554,66 +540,149 @@ function docker_run_image
   declare server
   declare retval
   declare elapsed
+  declare dockerdir
+  declare lines
+  declare dockerversion
+  declare -a dockplat
 
-  dockerfilerev=$(docker_getfilerev)
+  big_console_header "Docker Image Creation"
+  start_clock
 
-  baseimagename="yetus/${PROJECT_NAME}:${dockerfilerev}"
+  if [[ -n "${DOCKER_PLATFORM}" ]]; then
+    dockplat=('--platform' "${DOCKER_PLATFORM}")
+  fi
 
-  # make a base image, if it isn't available
-  big_console_header "Building base image: ${baseimagename}"
-  start_clock
-  dockercmd build \
-    -t "${baseimagename}" \
-    "${PATCH_DIR}/precommit/test-patch-docker"
-  retval=$?
+  if [[ -n "${DOCKER_TAG}" ]]; then
+    # pull the base image from the provided docker tag
 
-  #shellcheck disable=SC2046
-  elapsed=$(clock_display $(stop_clock))
+    if ! dockercmd pull "${dockplat[@]}" "${DOCKER_TAG}"; then
+      yetus_error "ERROR: Docker failed to pull ${DOCKER_TAG}."
+      add_vote_table -1 docker "Docker failed to pull ${DOCKER_TAG}."
+      bugsystem_finalreport 1
+      cleanup_and_exit 1
+    fi
+    baseimagename=${DOCKER_TAG}
 
-  echo ""
-  echo "Total Elapsed time: ${elapsed}"
-  echo ""
+  elif [[ -n ${DOCKERFILE} && -f ${DOCKERFILE} ]]; then
 
-  if [[ ${retval} != 0 ]]; then
-    yetus_error "ERROR: Docker failed to build image."
-    add_vote_table -1 docker "Docker failed to build ${baseimagename}."
-    bugsystem_finalreport 1
-    cleanup_and_exit 1
+    # make a base image. if it is available/cached, this will go quick
+    dockerdir=$(dirname "${DOCKERFILE}")
+
+    pushd "${dockerdir}" >/dev/null || return 1
+    # grab the git commit sha if this is part of a git repo, even if
+    # it is part of another git repo other than the one being tested!
+    gitfilerev=$("${GIT}" log -n 1 --pretty=format:%h -- "${DOCKERFILE}" 2>/dev/null)
+    if [[ -z ${gitfilerev} ]]; then
+      gitfilerev=$(date "+%F")
+      gitfilerev="date${gitfilerev}"
+    fi
+
+    # we will need this for reporting later
+    baseimagename="yetus/${PROJECT_NAME}:${gitfilerev}"
+
+    # create a new Dockerfile in the patchdir to actually use to
+    # build with, stripping everything after the cut here line
+    # (if it exists)
+    lines=$("${AWK}" '/YETUS CUT HERE/ {print FNR; exit}' "${DOCKERFILE}")
+
+    buildfile="${PATCH_DIR}/Dockerfile"
+    if [[ "${DOCKER_VERSION[0]}" -lt 18 ]] && [[ ${lines} -gt 0 ]]; then
+
+      # versions less than 18 don't support having the
+      # Dockerfile outside of the build context. Let's fall back to
+      # pre-YETUS-723 behavior and put the re-written Dockerfile
+      # outside of the source tree rather than go through a lot of
+      # machinations.  This means COPY, ADD, etc do not work, but
+      # whatever
+
+      popd >/dev/null || return 1
+      buildfile="${PATCH_DIR}/test-patch-docker/Dockerfile"
+      dockerdir="${PATCH_DIR}/test-patch-docker"
+      mkdir -p "${dockerdir}"
+      pushd "${PATCH_DIR}/test-patch-docker" >/dev/null || return 1
+    fi
+
+    (
+      if [[ -z "${lines}" ]]; then
+        cat "${DOCKERFILE}"
+      else
+        head -n "${lines}" "${DOCKERFILE}"
+      fi
+    ) > "${buildfile}"
+
+    if [[ "${DOCKER_VERSION[0]}" -lt 18 ]] && [[ ${lines} -gt 0 ]]; then
+      # Need to put our re-constructed Dockerfile in a place
+      # where it can be referenced in the output post-run
+      cp -p "${buildfile}" "${PATCH_DIR}/Dockerfile"
+    fi
+
+    if ! dockercmd build \
+          "${dockplat[@]}" \
+          --label org.apache.yetus=\"\" \
+          --label org.apache.yetus.testpatch.project="${PROJECT_NAME}" \
+          --tag "${baseimagename}" \
+          -f "${buildfile}" \
+          "${dockerdir}"; then
+      popd >/dev/null || return 1
+      yetus_error "ERROR: Docker failed to build ${baseimagename}."
+      add_vote_table -1 docker "Docker failed to build ${baseimagename}."
+      bugsystem_finalreport 1
+      cleanup_and_exit 1
+    fi
+    popd >/dev/null || return 1
   fi
 
-  big_console_header "Building ${BUILDMODE} image: ${patchimagename}"
-  start_clock
+  echo "Building run-specific image ${patchimagename}"
+
+  # create a directory from scratch so that our
+  # build context is tightly controlled
+  randir=${PATCH_DIR}/${RANDOM}
+  mkdir -p "${randir}"
+  pushd "${randir}" >/dev/null || return 1
+  cp -p "${BINDIR}"/test-patch-docker/Dockerfile.patchspecific \
+     "${randir}"/Dockerfile
+  cp -p "${BINDIR}"/test-patch-docker/launch-test-patch.sh \
+     "${randir}"
+
+
+  for lines in "${USER_PARAMS[@]}"; do
+    if [[ ${lines} != '--docker' ]]; then
+      echo "${lines}" >> "${randir}/user_params.txt"
+    fi
+  done
+
   # using the base image, make one that is patch specific
   dockercmd build \
-    -t "${patchimagename}" \
-    - <<PatchSpecificDocker
-FROM ${baseimagename}
-LABEL org.apache.yetus=""
-LABEL org.apache.yetus.testpatch.patch="tp-${DOCKER_ID}"
-LABEL org.apache.yetus.testpatch.project=${PROJECT_NAME}
-RUN groupadd --non-unique -g ${GROUP_ID} ${USER_NAME} || true
-RUN useradd -g ${GROUP_ID} -u ${USER_ID} -m ${USER_NAME} || true
-RUN chown -R ${USER_NAME} /home/${USER_NAME} || true
-ENV HOME /home/${USER_NAME}
-USER ${USER_NAME}
-PatchSpecificDocker
-
+    "${dockplat[@]}" \
+    --no-cache \
+    --build-arg baseimagename="${baseimagename}" \
+    --build-arg GROUP_ID="${GROUP_ID}" \
+    --build-arg USER_ID="${USER_ID}" \
+    --build-arg USER_NAME="${USER_NAME}" \
+    --label org.apache.yetus=\"\" \
+    --label org.apache.yetus.testpatch.patch="tp-${DOCKER_ID}" \
+    --label org.apache.yetus.testpatch.project="${PROJECT_NAME}" \
+    --tag "${patchimagename}" \
+    "${randir}" >/dev/null
   retval=$?
+  popd >/dev/null || retun 1
 
-  #shellcheck disable=SC2046
-  elapsed=$(clock_display $(stop_clock))
-
-  echo ""
-  echo "Total Elapsed time: ${elapsed}"
-  echo ""
+  rm -rf "${randir}"
 
   if [[ ${retval} != 0 ]]; then
-    yetus_error "ERROR: Docker failed to build image."
-    add_vote_table -1 docker "Docker failed to build ${patchimagename}."
+    yetus_error "ERROR: Docker failed to build run-specific ${patchimagename}."
+    add_vote_table -1 docker "Docker failed to build run-specific ${patchimagename}}."
     bugsystem_finalreport 1
     cleanup_and_exit 1
   fi
 
+  #shellcheck disable=SC2046
+  elapsed=$(clock_display $(stop_clock))
+
+  echo ""
+  echo "Total elapsed build time: ${elapsed}"
+  echo ""
+
   if [[ "${DOCKER_ENABLE_PRIVILEGED}" = true ]]; then
     DOCKER_EXTRAARGS+=("--privileged")
   fi
@@ -627,17 +696,22 @@ PatchSpecificDocker
 
   dockerversion="Client=${client} Server=${server}"
 
-
   # make the kernel prefer to kill us if we run out of RAM
   DOCKER_EXTRAARGS+=("--oom-score-adj" "500")
 
   DOCKER_EXTRAARGS+=("--cidfile=${PATCH_DIR}/cidfile")
+
+  if [[ "${DOCKER_IN_DOCKER}" == true ]]; then
+    if [[ -e "${DOCKER_SOCKET}" ]]; then
+      DOCKER_EXTRAARGS+=(-v "${DOCKER_SOCKET}:/var/run/docker.sock")
+    fi
+  fi
+
   DOCKER_EXTRAARGS+=(-v "${PWD}:/testptch/${PROJECT_NAME}")
   DOCKER_EXTRAARGS+=(-u "${USER_NAME}")
   DOCKER_EXTRAARGS+=(-w "/testptch/${PROJECT_NAME}")
   DOCKER_EXTRAARGS+=("--env=BASEDIR=/testptch/${PROJECT_NAME}")
-  DOCKER_EXTRAARGS+=("--env=DOCKER_VERSION=${dockerversion} Image:${baseimagename}")
-  DOCKER_EXTRAARGS+=("--env=TESTPATCHMODE=${TESTPATCHMODE}")
+  DOCKER_EXTRAARGS+=("--env=DOCKER_VERSION_STR=${dockerversion}")
 
   docker_do_env_adds
 
@@ -648,19 +722,25 @@ PatchSpecificDocker
 
   if [[ ${PATCH_DIR} =~ ^/ ]]; then
     dockercmd run --rm=true -i \
+      "${dockplat[@]}" \
       "${DOCKER_EXTRAARGS[@]}" \
       -v "${PATCH_DIR}:/testptch/patchprocess" \
       --env=PATCH_DIR=/testptch/patchprocess \
       "${patchimagename}" &
   else
     dockercmd run --rm=true -i \
+      "${dockplat[@]}" \
       "${DOCKER_EXTRAARGS[@]}" \
       --env=PATCH_DIR="${PATCH_DIR}" \
       "${patchimagename}" &
   fi
-
   wait ${!}
-  cleanup_and_exit $?
+  retval=$?
+
+  printf '\n\n'
+  echo "Cleaning up docker image used for testing."
+  dockercmd rmi "${patchimagename}" > /dev/null
+  cleanup_and_exit ${retval}
 }
 
 ## @description  docker kill the container on SIGTERM
@@ -686,9 +766,52 @@ function docker_signal_handler
 ## @param        args
 function docker_handler
 {
-  PATCH_DIR=$(relative_dir "${PATCH_DIR}")
+  declare plugin
+  declare person
+
+  ## @description  strip paths for display
+  ## @audience     private
+  ## @stability    evolving
+  function strippaths {
+    declare p=$1
+    declare d
+    for d in "${BASEDIR}" "${PATCH_DIR}" "${BINDIR}"; do
+      p=$(yetus_relative_dir "${d}" "${p}")
+    done
+    echo "${p}"
+  }
+
 
-  docker_cleanup
   determine_user
+
+  # need to call this explicitly
+  console_docker_support
+
+  for plugin in ${PROJECT_NAME} ${BUILDTOOL} ${BUGSYSTEMS} ${TESTTYPES} ${TESTFORMATS}; do
+    if declare -f "${plugin}_docker_support" >/dev/null; then
+      "${plugin}_docker_support"
+    fi
+  done
+
+  if [[ -n "${BUILD_URL}" ]]; then
+    USER_PARAMS+=("--build-url=${BUILD_URL}")
+  fi
+
+  if [[ -f "${PERSONALITY}" ]]; then
+    person=$(strippaths "${PERSONALITY}")
+    USER_PARAMS+=("--tpperson=${person}")
+  fi
+
+  USER_PARAMS+=("--tpglobaltimer=${GLOBALTIMER}")
+  USER_PARAMS+=("--tpreexectimer=${TIMER}")
+  USER_PARAMS+=("--tpinstance=${INSTANCE}")
+  USER_PARAMS+=("--plugins=${ENABLED_PLUGINS// /,}")
+
+  #shellcheck disable=SC2164
+  cd "${BASEDIR}"
+
+  PATCH_DIR=$(yetus_relative_dir "${BASEDIR}" "${PATCH_DIR}")
+
+  docker_cleanup
   docker_run_image
 }
diff --git a/precommit/src/main/shell/test-patch-docker/Dockerfile b/precommit/src/main/shell/test-patch-docker/Dockerfile
index 981cb5f..8ad2631 100644
--- a/precommit/src/main/shell/test-patch-docker/Dockerfile
+++ b/precommit/src/main/shell/test-patch-docker/Dockerfile
@@ -39,6 +39,8 @@ ENV DEBCONF_TERSE true
 # some git repos need ssh-client so do it too
 ######
 RUN apt-get -q update && apt-get -q install --no-install-recommends -y \
+    apt-transport-https \
+    ca-certificates \
     curl \
     git \
     locales \
@@ -99,6 +101,18 @@ RUN apt-get -q update && apt-get -q install --no-install-recommends -y cmake \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
 
+###
+# Install docker
+###
+RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+RUN add-apt-repository -y \
+   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+   $(lsb_release -cs) \
+   stable"
+RUN apt-get -q update && apt-get -q install --no-install-recommends -y docker-ce \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
 ######
 # Install findbugs
 ######
@@ -140,7 +154,8 @@ RUN apt-get -q update && apt-get -q install --no-install-recommends -y \
     python3-dev \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
-RUN pip3 install -v pylint==2.1.1
+#hadolint ignore=DL3013
+RUN pip3 install -v pylint==2.1.1 docker-compose
 RUN mv /usr/local/bin/pylint /usr/local/bin/pylint3
 
 ######
diff --git a/precommit/src/main/shell/test-patch-docker/Dockerfile-endstub b/precommit/src/main/shell/test-patch-docker/Dockerfile.patchspecific
similarity index 62%
rename from precommit/src/main/shell/test-patch-docker/Dockerfile-endstub
rename to precommit/src/main/shell/test-patch-docker/Dockerfile.patchspecific
index 321fafe..5a7ac38 100644
--- a/precommit/src/main/shell/test-patch-docker/Dockerfile-endstub
+++ b/precommit/src/main/shell/test-patch-docker/Dockerfile.patchspecific
@@ -14,8 +14,24 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-COPY launch-test-patch.sh /testptch/launch-test-patch.sh
-RUN mkdir /testptch/extras
+ARG baseimagename
+# hadolint ignore=DL3006
+FROM ${baseimagename}
+
+ARG USER_NAME
+ARG GROUP_ID
+ARG USER_ID
+
+RUN groupadd --non-unique -g "${GROUP_ID}" "${USER_NAME}" || true
+RUN useradd -g "${GROUP_ID}" -u "${USER_ID}" -m "${USER_NAME}" || true
+RUN groupadd -g docker "${USER_NAME}" || true
+RUN mkdir -p /testptch/extras
 RUN chmod a+rwx /testptch/extras
+COPY user_params.txt /testptch/user_params.txt
+COPY launch-test-patch.sh /testptch/launch-test-patch.sh
+RUN chmod a+rwx /testptch/user_params.txt
 RUN chmod a+rx /testptch/launch-test-patch.sh
+RUN chown -R "${USER_NAME}" "/home/${USER_NAME}" || true
+ENV HOME "/home/${USER_NAME}"
+USER ${USER_NAME}
 CMD ["/testptch/launch-test-patch.sh"]
\ No newline at end of file
diff --git a/precommit/src/main/shell/test-patch-docker/launch-test-patch.sh b/precommit/src/main/shell/test-patch-docker/launch-test-patch.sh
index b170cd2..b9fee89 100755
--- a/precommit/src/main/shell/test-patch-docker/launch-test-patch.sh
+++ b/precommit/src/main/shell/test-patch-docker/launch-test-patch.sh
@@ -16,15 +16,28 @@
 
 declare -a OVERWRITEARGS
 
-OVERWRITEARGS=("--reexec")
-OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--dockermode")
-OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--basedir=${BASEDIR}")
+# yetuslib is meant to be completely standalone.  This means
+# adding it as a common function library here is a-ok
+
+#shellcheck disable=SC1090
+source "${PATCH_DIR}/precommit/core.d/00-yetuslib.sh"
+
+if ! yetus_file_to_array OVERWRITEARGS /testptch/user_params.txt; then
+  yetus_error "ERROR: Cannot read user parameters file. Exiting."
+  exit 1
+fi
+
+# do not want this archived
+rm /testptch/user_params.txt
+
+OVERWRITEARGS+=("--reexec")
+OVERWRITEARGS+=("--dockermode")
+OVERWRITEARGS+=("--basedir=${BASEDIR}")
 
 cd "${BASEDIR}" || exit 1
 
-if [[ -n ${JAVA_HOME}
-  && ! -d ${JAVA_HOME} ]]; then
-  echo "JAVA_HOME: ${JAVA_HOME} does not exist. Dockermode: attempting to switch to another." 1>&2
+if [[ -n ${JAVA_HOME} && ! -d ${JAVA_HOME} ]]; then
+  yetus_error "WARNING: JAVA_HOME=${JAVA_HOME} does not exist. Dockermode: attempting to switch to another." 1>&2
   JAVA_HOME=""
 fi
 
@@ -32,24 +45,16 @@ if [[ -z ${JAVA_HOME} ]]; then
   JAVA_HOME=$(find /usr/lib/jvm/ -name "java-*" -type d | tail -1)
   export JAVA_HOME
   if [[ -n "${JAVA_HOME}" ]]; then
-    OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--java-home=${JAVA_HOME}")
-    echo "Setting ${JAVA_HOME} as the JAVA_HOME."
+    OVERWRITEARGS+=("--java-home=${JAVA_HOME}")
+    yetus_error "WARNING: Setting ${JAVA_HOME} as the JAVA_HOME."
   fi
 fi
 
-# Avoid out of memory errors in builds
-MAVEN_OPTS=${MAVEN_OPTS:-"-Xms256m -Xmx1g"}
-export MAVEN_OPTS
-
-# strip out --docker param to prevent re-exec again
-TESTPATCHMODE=${TESTPATCHMODE/--docker }
-TESTPATCHMODE=${TESTPATCHMODE%--docker}
-
-PATCH_DIR=$(cd -P -- "${PATCH_DIR}" >/dev/null && pwd -P)
-OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--patch-dir=${PATCH_DIR}")
-OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--user-plugins=${PATCH_DIR}/precommit/user-plugins")
+PATCH_DIR=$(yetus_abs "${PATCH_DIR}")
+OVERWRITEARGS+=("--patch-dir=${PATCH_DIR}")
+OVERWRITEARGS+=("--user-plugins=${PATCH_DIR}/precommit/user-plugins")
 if [[ -f "${PATCH_DIR}/precommit/unit_test_filter_file.txt" ]]; then
-  OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--unit-test-filter-file=${PATCH_DIR}/precommit/unit_test_filter_file.txt")
+  OVERWRITEARGS+=("--unit-test-filter-file=${PATCH_DIR}/precommit/unit_test_filter_file.txt")
 fi
 
 # if patch system is generic, then it's either a local
@@ -58,23 +63,13 @@ fi
 # test-patch where to find it.
 if [[ "${PATCH_SYSTEM}" = generic ]]; then
   cp -p "${PATCH_DIR}/patch" /testptch/extras/patch
-  OVERWRITEARGS=("${OVERWRITEARGS[@]}" "/testptch/extras/patch")
-fi
-
-if [[ -f /testptch/console.txt ]]; then
-  OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--console-report-file=/testptch/console.txt")
-fi
-
-if [[ -f /testptch/brief.txt ]]; then
-  OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--brief-report-file=/testptch/brief.txt")
-fi
-
-if [[ -f /testptch/report.htm ]]; then
-  OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--html-report-file=/testptch/report.htm")
+  OVERWRITEARGS+=("/testptch/extras/patch")
 fi
 
 cd "${PATCH_DIR}/precommit/" || exit 1
-#shellcheck disable=SC2086
-"${PATCH_DIR}/precommit/test-patch.sh" \
-   ${TESTPATCHMODE} \
-  "${OVERWRITEARGS[@]}"
+
+if [[ "${YETUS_DOCKER_BASH_DEBUG}" == true ]]; then
+  exec bash -x "${PATCH_DIR}/precommit/test-patch.sh" "${OVERWRITEARGS[@]}"
+else
+  exec "${PATCH_DIR}/precommit/test-patch.sh" "${OVERWRITEARGS[@]}"
+fi
\ No newline at end of file
diff --git a/precommit/src/main/shell/test-patch.d/ant.sh b/precommit/src/main/shell/test-patch.d/ant.sh
index 9dd8240..31a5e66 100755
--- a/precommit/src/main/shell/test-patch.d/ant.sh
+++ b/precommit/src/main/shell/test-patch.d/ant.sh
@@ -25,7 +25,7 @@ fi
 
 add_build_tool ant
 
-declare -a ANT_ARGS=("-noinput")
+declare -a YETUS_ANT_ARGS=("-noinput")
 
 function ant_usage
 {
@@ -46,7 +46,7 @@ function ant_parse_args
 
   # if we requested offline, pass that to ant
   if [[ ${OFFLINE} == "true" ]]; then
-    ANT_ARGS=("${ANT_ARGS[@]}" '-Doffline=')
+    YETUS_ANT_ARGS+=('-Doffline=')
   fi
 }
 
@@ -95,7 +95,7 @@ function ant_buildfile
 
 function ant_executor
 {
-  echo "${ANT}" "${ANT_ARGS[@]}"
+  echo "${ANT}" "${YETUS_ANT_ARGS[@]}"
 }
 
 function ant_modules_worker
@@ -215,5 +215,7 @@ function ant_builtin_personality_file_tests
 
 function ant_docker_support
 {
-  DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS}" "-v" "${HOME}/.ivy2:/home/${USER_NAME}/.ivy2")
+  DOCKER_EXTRAARGS+=("-v" "${HOME}/.ivy2:/home/${USER_NAME}/.ivy2")
+  add_docker_env ANT_OPTS
+  add_docker_env ANT_ARGS
 }
diff --git a/precommit/src/main/shell/test-patch.d/briefreport.sh b/precommit/src/main/shell/test-patch.d/briefreport.sh
index caeaaf0..2554edb 100755
--- a/precommit/src/main/shell/test-patch.d/briefreport.sh
+++ b/precommit/src/main/shell/test-patch.d/briefreport.sh
@@ -69,7 +69,8 @@ function briefreport_parse_args
 function briefreport_docker_support
 {
   if [[ -n ${BRIEFOUT_REPORTFILE} ]]; then
-    DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${BRIEFOUT_REPORTFILE}:/testptch/brief.txt")
+    DOCKER_EXTRAARGS+=("-v" "${BRIEFOUT_REPORTFILE}:/testptch/brief.txt")
+    USER_PARAMS+=("--brief-report-file=/testptch/brief.txt")
   fi
 }
 
@@ -219,7 +220,8 @@ function briefreport_finalreport
 
   i=0
   until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do
-    if [[ "${TP_FOOTER_TABLE[${i}]}" =~ \@\@BASE\@\@ ]]; then
+    if [[ "${TP_FOOTER_TABLE[${i}]}" =~ \@\@BASE\@\@ ]] \
+       && [[ ! "${TP_FOOTER_TABLE[${i}]}" =~ Dockerfile ]]; then
       havelogs=true
       break
     fi
diff --git a/precommit/src/main/shell/test-patch.d/gradle.sh b/precommit/src/main/shell/test-patch.d/gradle.sh
index 6c580ba..64e9714 100755
--- a/precommit/src/main/shell/test-patch.d/gradle.sh
+++ b/precommit/src/main/shell/test-patch.d/gradle.sh
@@ -290,5 +290,5 @@ function gradle_builtin_personality_file_tests
 
 function gradle_docker_support
 {
-  DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${HOME}/.gradle:/home/${USER_NAME}/.gradle")
+  DOCKER_EXTRAARGS+=("-v" "${HOME}/.gradle:/home/${USER_NAME}/.gradle")
 }
diff --git a/precommit/src/main/shell/test-patch.d/htmlout.sh b/precommit/src/main/shell/test-patch.d/htmlout.sh
index ef44cb0..9660001 100755
--- a/precommit/src/main/shell/test-patch.d/htmlout.sh
+++ b/precommit/src/main/shell/test-patch.d/htmlout.sh
@@ -59,7 +59,8 @@ function htmlout_parse_args
 function htmlout_docker_support
 {
   if [[ -n ${HTMLOUT_REPORTFILE} ]]; then
-    DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${HTMLOUT_REPORTFILE}:/testptch/report.htm")
+    DOCKER_EXTRAARGS+=("-v" "${HTMLOUT_REPORTFILE}:/testptch/report.htm")
+    USER_PARAMS+=("--html-report-file=/testptch/report.htm")
   fi
 }
 
diff --git a/precommit/src/main/shell/test-patch.d/maven.sh b/precommit/src/main/shell/test-patch.d/maven.sh
index 7741254..59f6a66 100755
--- a/precommit/src/main/shell/test-patch.d/maven.sh
+++ b/precommit/src/main/shell/test-patch.d/maven.sh
@@ -638,17 +638,19 @@ function maven_precompile
   return 0
 }
 
-## @description  set volumes as appropriate for maven
+## @description  set volumes and options as appropriate for maven
 ## @audience     private
 ## @stability    evolving
 ## @replaceable  yes
 function maven_docker_support
 {
-  DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${HOME}/.m2:/home/${USER_NAME}/.m2")
+  DOCKER_EXTRAARGS+=("-v" "${HOME}/.m2:/home/${USER_NAME}/.m2")
 
   if [[ ${MAVEN_CUSTOM_REPOS} = true ]]; then
-    DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${MAVEN_CUSTOM_REPOS_DIR}:${MAVEN_CUSTOM_REPOS_DIR}")
+    DOCKER_EXTRAARGS+=("-v" "${MAVEN_CUSTOM_REPOS_DIR}:${MAVEN_CUSTOM_REPOS_DIR}")
   fi
+
+  add_docker_env MAVEN_OPTS
 }
 
 ## @description  worker for maven reordering. MAVEN_DEP_LOG is set to the log file name
diff --git a/precommit/src/main/shell/test-patch.sh b/precommit/src/main/shell/test-patch.sh
index 9a78057..8e62afa 100755
--- a/precommit/src/main/shell/test-patch.sh
+++ b/precommit/src/main/shell/test-patch.sh
@@ -669,27 +669,6 @@ function echo_and_redirect
 
     yetus_run_and_redirect "${logfile}" "${@}"
   fi
-
-}
-
-## @description is a given directory relative to BASEDIR?
-## @audience    public
-## @stability   stable
-## @replaceable yes
-## @param       path
-## @return      1 - no, path
-## @return      0 - yes, path - BASEDIR
-function relative_dir
-{
-  local p=${1#${BASEDIR}}
-
-  if [[ ${#p} -eq ${#1} ]]; then
-    echo "${p}"
-    return 1
-  fi
-  p=${p#/}
-  echo "${p}"
-  return 0
 }
 
 ## @description  Print the usage information
@@ -1068,7 +1047,6 @@ function parse_args
 
   if [[ "${REEXECED}" = true
      && -f "${PATCH_DIR}/precommit/personality/provided.sh" ]]; then
-    REEXECPERSONALITY="${PERSONALITY}"
     PERSONALITY="${PATCH_DIR}/precommit/personality/provided.sh"
   fi
 }
@@ -1373,7 +1351,7 @@ function git_checkout
 
     # if PATCH_DIR is in BASEDIR, then we don't want
     # git wiping it out.
-    exemptdir=$(relative_dir "${PATCH_DIR}")
+    exemptdir=$(yetus_relative_dir "${BASEDIR}" "${PATCH_DIR}")
     if [[ $? == 1 ]]; then
       "${GIT}" clean -xdf
       status=$?
@@ -1633,10 +1611,6 @@ function apply_patch_file
 ## @replaceable  no
 function copytpbits
 {
-  declare dockerdir
-  declare dockfile
-  declare lines
-
   # we need to copy/consolidate all the bits that might have changed
   # that are considered part of test-patch.  This *might* break
   # things that do off-path includes, but there isn't much we can
@@ -1681,38 +1655,6 @@ function copytpbits
     cp -pr "${UNIT_TEST_FILTER_FILE}" "${PATCH_DIR}/precommit/unit_test_filter_file.txt"
   fi
 
-  if [[ -n ${DOCKERFILE}
-      && -f ${DOCKERFILE} ]]; then
-    yetus_debug "copying '${DOCKERFILE}' over to '${PATCH_DIR}/precommit/test-patch-docker/Dockerfile'"
-    dockerdir=$(dirname "${DOCKERFILE}")
-    dockfile=$(basename "${DOCKERFILE}")
-    pushd "${dockerdir}" >/dev/null || return 1
-    gitfilerev=$("${GIT}" log -n 1 --pretty=format:%h -- "${dockfile}" 2>/dev/null)
-    popd >/dev/null || return 1
-    if [[ -z ${gitfilerev} ]]; then
-      gitfilerev=$(date "+%F")
-      gitfilerev="date${gitfilerev}"
-    fi
-    (
-      echo "### YETUS_PRIVATE: dockerfile=${DOCKERFILE}"
-      echo "### YETUS_PRIVATE: gitrev=${gitfilerev}"
-      lines=$(${GREP} -n 'YETUS CUT HERE' "${DOCKERFILE}" | cut -f1 -d:)
-      if [[ -z "${lines}" ]]; then
-        cat "${DOCKERFILE}"
-      else
-        head -n "${lines}" "${DOCKERFILE}"
-      fi
-      # make sure we put some space between, just in case last
-      # line isn't an empty line or whatever
-      printf '\n\n'
-      echo "### YETUS_PRIVATE: start test-patch-bootstrap"
-      cat "${BINDIR}/test-patch-docker/Dockerfile-endstub"
-
-      printf '\n\n'
-    ) > "${PATCH_DIR}/precommit/test-patch-docker/Dockerfile"
-    DOCKERFILE="${PATCH_DIR}/precommit/test-patch-docker/Dockerfile"
-  fi
-
   popd >/dev/null || return 1
 }
 
@@ -1777,7 +1719,7 @@ function check_reexec
       "${PERSONALITY}" \
       "${USER_PLUGIN_DIR}" \
       "${DOCKERFILE}"; do
-    tpdir=$(relative_dir "${testdir}")
+    tpdir=$(yetus_relative_dir "${BASEDIR}" "${testdir}")
     # shellcheck disable=SC2181
     if [[ $? == 0
         && "${CHANGED_FILES[*]}" =~ ${tpdir} ]]; then
@@ -1821,40 +1763,12 @@ function check_reexec
   copytpbits
 
   if [[ ${DOCKERSUPPORT} == true ]]; then
-    # if we are doing docker, then we re-exec, but underneath the
-    # container
-
-    determine_user
 
-    # need to call this explicitly
-    console_docker_support
+    #if we are doing docker, then we re-exec, but underneath the
+    #container
 
-    for plugin in ${PROJECT_NAME} ${BUILDTOOL} ${BUGSYSTEMS} ${TESTTYPES} ${TESTFORMATS}; do
-      if declare -f "${plugin}_docker_support" >/dev/null; then
-        "${plugin}_docker_support"
-      fi
-    done
-
-    TESTPATCHMODE="${USER_PARAMS[*]}"
-    if [[ -n "${BUILD_URL}" ]]; then
-      TESTPATCHMODE="--build-url=${BUILD_URL} ${TESTPATCHMODE}"
-    fi
-
-    if [[ -f "${PERSONALITY}" ]]; then
-      TESTPATCHMODE="--tpperson=${PERSONALITY} ${TESTPATCHMODE}"
-    fi
-
-    TESTPATCHMODE="--tpglobaltimer=${GLOBALTIMER} ${TESTPATCHMODE}"
-    TESTPATCHMODE="--tpreexectimer=${TIMER} ${TESTPATCHMODE}"
-    TESTPATCHMODE="--tpinstance=${INSTANCE} ${TESTPATCHMODE}"
-    TESTPATCHMODE="--plugins=${ENABLED_PLUGINS// /,} ${TESTPATCHMODE}"
-    TESTPATCHMODE=" ${TESTPATCHMODE}"
-    export TESTPATCHMODE
-
-    #shellcheck disable=SC2164
-    cd "${BASEDIR}"
-    #shellcheck disable=SC2093
     docker_handler
+    exit $?
   else
 
     # if we aren't doing docker, then just call ourselves
@@ -2457,14 +2371,18 @@ function cleanup_and_exit
       # there is no need to move it since we assume that
       # Jenkins or whatever already knows where it is at
       # since it told us to put it there!
-      relative_dir "${PATCH_DIR}" >/dev/null
+      yetus_relative_dir "${BASEDIR}" "${PATCH_DIR}" >/dev/null
       if [[ $? == 1 ]]; then
         yetus_debug "mv ${PATCH_DIR} ${BASEDIR}"
         mv "${PATCH_DIR}" "${BASEDIR}"
       fi
     fi
   fi
-  big_console_header "Finished build."
+
+  # docker mode will print this after exit
+  if [[ "${DOCKERMODE}" == false ]]; then
+    big_console_header "Finished build."
+  fi
 
   # shellcheck disable=SC2086
   exit ${result}


Mime
View raw message