Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C4F2C200B20 for ; Wed, 11 May 2016 23:41:26 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C3AA0160A17; Wed, 11 May 2016 21:41:26 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 83C8B160A23 for ; Wed, 11 May 2016 23:41:24 +0200 (CEST) Received: (qmail 8237 invoked by uid 500); 11 May 2016 21:41:23 -0000 Mailing-List: contact commits-help@yetus.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@yetus.apache.org Delivered-To: mailing list commits@yetus.apache.org Received: (qmail 7997 invoked by uid 99); 11 May 2016 21:41:23 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2016 21:41:23 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 7C964E38E3; Wed, 11 May 2016 21:41:23 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: sekikn@apache.org To: commits@yetus.apache.org Date: Wed, 11 May 2016 21:41:34 -0000 Message-Id: <896d70736f854a20aa0f755d2900ae17@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [12/20] yetus git commit: YETUS-156. [Umbrella] build driver archived-at: Wed, 11 May 2016 21:41:26 -0000 YETUS-156. [Umbrella] build driver This is a rebased and squashed commit of the following issues: YETUS-347. change docker handler to use arrays YETUS-348. add ability to write console report to a file YETUS-349. core engine changes for build driver YETUS-350. plug-in changes for build driver YETUS-353. runtime wrapper for build driver YETUS-351. documentation changes for build driver YETUS-352. hadoop and hbase personality changes for build driver YETUS-354. findbugs in build driver has problems YETUS-371. build driver: speed up checkstyle YETUS-357. build driver for hadoop shouldn't union unit tests YETUS-372. build driver: unit tests still mentions patches YETUS-373. build driver: maven install should be smarter YETUS-356. build driver: author plug-in still mentions patches YETUS-356. build driver: author plug-in still mentions patches (addendum) YETUS-374. need a way to ignore results from a list of tests YETUS-375. build driver: maven install actually needs to be dumber YETUS-376. add ability to write report to an HTML formatted file YETUS-384. In build driver mode, some plugins have misleading status text YETUS-385. test plugin for author tag detection should have a way to opt-out of specific instances YETUS-378. flink, hbase, jmeter personalities custom tests need to be made qbt aware YETUS-386. document --html-report-file and clean up htmlout's comments Signed-off-by: Sean Busbey Project: http://git-wip-us.apache.org/repos/asf/yetus/repo Commit: http://git-wip-us.apache.org/repos/asf/yetus/commit/612a0f2d Tree: http://git-wip-us.apache.org/repos/asf/yetus/tree/612a0f2d Diff: http://git-wip-us.apache.org/repos/asf/yetus/diff/612a0f2d Branch: refs/heads/YETUS-379 Commit: 612a0f2d12af3c6713b04c29ba6406f17b29f0fa Parents: e3ea8a4 Author: Allen Wittenauer Authored: Tue Mar 29 13:17:19 2016 -0700 Committer: Allen Wittenauer Committed: Mon Apr 25 08:00:09 2016 -0700 ---------------------------------------------------------------------- .../in-progress/precommit-advanced.md | 5 + .../in-progress/precommit-buildtools.md | 2 +- .../documentation/in-progress/precommit-qbt.md | 64 +++++ build.sh | 3 +- precommit/core.d/00-yetuslib.sh | 2 +- precommit/core.d/01-common.sh | 22 ++ precommit/core.d/builtin-bugsystem.sh | 10 + precommit/core.d/docker.sh | 24 +- precommit/personality/flink.sh | 61 +++-- precommit/personality/hadoop.sh | 14 +- precommit/personality/hbase.sh | 167 +++++++++++-- precommit/personality/jmeter.sh | 3 + .../test-patch-docker/launch-test-patch.sh | 29 ++- precommit/test-patch.d/ant.sh | 2 +- precommit/test-patch.d/asflicense.sh | 8 +- precommit/test-patch.d/author.sh | 118 ++++++++-- precommit/test-patch.d/autoconf.sh | 14 +- precommit/test-patch.d/checkstyle.sh | 46 ++-- precommit/test-patch.d/cmake.sh | 4 +- precommit/test-patch.d/findbugs.sh | 29 ++- precommit/test-patch.d/gradle.sh | 6 +- precommit/test-patch.d/htmlout.sh | 229 ++++++++++++++++++ precommit/test-patch.d/maven.sh | 43 +++- precommit/test-patch.d/perlcritic.sh | 12 +- precommit/test-patch.d/pylint.sh | 16 +- precommit/test-patch.d/rubocop.sh | 12 +- precommit/test-patch.d/ruby-lint.sh | 12 +- precommit/test-patch.d/shellcheck.sh | 8 +- precommit/test-patch.d/shelldocs.sh | 8 +- precommit/test-patch.d/test4tests.sh | 4 + precommit/test-patch.d/whitespace.sh | 38 ++- precommit/test-patch.d/xml.sh | 6 +- precommit/test-patch.sh | 232 +++++++++++++------ 33 files changed, 983 insertions(+), 270 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/asf-site-src/source/documentation/in-progress/precommit-advanced.md ---------------------------------------------------------------------- diff --git a/asf-site-src/source/documentation/in-progress/precommit-advanced.md b/asf-site-src/source/documentation/in-progress/precommit-advanced.md index d64392a..953b44e 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-advanced.md +++ b/asf-site-src/source/documentation/in-progress/precommit-advanced.md @@ -80,6 +80,9 @@ Similarly, there are other functions that may be defined during the test-patch r * pluginname\_initialize - After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc. +* pluginname\_docker\_support + - Perform any necessary setup to configure Docker support for the given plugin. Typically this means adding parameters to the docker run command line via adding to the DOCKER\_EXTRAARGS array. + * pluginname\_precheck - executed prior to the patch being applied but after the git repository is setup. Returning a fail status here will exit test-patch. @@ -253,6 +256,8 @@ There are a handful of extremely important system variables that make life easie * CHANGED\_MODULES[@] is an array of all modules that house all of the CHANGED\_FILES[@]. Be aware that the root of the source tree is reported as '.'. +* DOCKER\_EXTRAARGS[@] is an array of command line arguments to apply to the `docker run` command. + * GITHUB\_REPO is to help test-patch when talking to Github. If test-patch is given just a number on the command line, it will default to using this repo to determine the pull request. * JIRA\_ISSUE\_RE is to help test-patch when talking to JIRA. It helps determine if the given project is appropriate for the given JIRA issue. http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/asf-site-src/source/documentation/in-progress/precommit-buildtools.md ---------------------------------------------------------------------- diff --git a/asf-site-src/source/documentation/in-progress/precommit-buildtools.md b/asf-site-src/source/documentation/in-progress/precommit-buildtools.md index e800135..bf530ba 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-buildtools.md +++ b/asf-site-src/source/documentation/in-progress/precommit-buildtools.md @@ -102,7 +102,7 @@ For example, the gradle build tool does not have a standard way to execute check * pluginname\_docker\_support - - If this build tool requires extra settings on the `docker run` command line, this function should be defined and write those options into a file called `${PATCH_DIR}/buildtool-docker-params.txt`. This is particularly useful for things like mounting volumes for repository caches. + - If this build tool requires extra settings on the `docker run` command line, this function should be defined and add those options into an array called `${DOCKER_EXTRAARGS[@]}`. This is particularly useful for things like mounting volumes for repository caches. **WARNING**: Be aware that directories that do not exist MAY be created by root by Docker itself under certain conditions. It is HIGHLY recommend that `pluginname_initialize` be used to create the necessary directories prior to be used in the `docker run` command. http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/asf-site-src/source/documentation/in-progress/precommit-qbt.md ---------------------------------------------------------------------- diff --git a/asf-site-src/source/documentation/in-progress/precommit-qbt.md b/asf-site-src/source/documentation/in-progress/precommit-qbt.md new file mode 100644 index 0000000..3e8f578 --- /dev/null +++ b/asf-site-src/source/documentation/in-progress/precommit-qbt.md @@ -0,0 +1,64 @@ + + +qbt +=== + +`qbt` is a command to execute test-patch without a patch. It uses +the same plug-ins and the same options as test-patch. The only +difference is that no patch file, location, etc should be supplied. +It is meant to be a way to easily get test-patch's output on your +current source tree. It is suitable to be run as a regularly +scheduled build as part of your overall development strategy. + +When using an automation tool, it may be useful to use the +`--console-report-file` option to send the summary email to a +file. This can then be used with systems like Jenkin's +email-ext plug-in to send the output as an emailed report: + +``` +${FILE,path=""} +``` + +For something a bit more structured, there is also the `--html-report-file` +option. Using this output, again with Jenkins' email-ext plug-in, it is +possible to build some very nice looking output that is easily customized: + +``` + + + + + +

See the Jenkins Build for more information.

+

${CHANGES, format="

[%d] (%a) %m
"}

+

+${FILE,path=""} + +``` + +NOTE: Be aware that ASF mailing lists do not allow HTML formatted email. http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/build.sh ---------------------------------------------------------------------- diff --git a/build.sh b/build.sh index 9fecbec..d5f3904 100755 --- a/build.sh +++ b/build.sh @@ -169,6 +169,7 @@ cp -r shelldocs "${bin_tarball}/lib/" cp -r release-doc-maker "${bin_tarball}/lib/" cp -r precommit "${bin_tarball}/lib/" +ln -s test-patch.sh "${bin_tarball}/lib/precommit/qbt.sh" mkdir -p "${bin_tarball}/bin" @@ -204,7 +205,7 @@ exec "\$(dirname -- "\${BASH_SOURCE-0}")/../lib/release-doc-maker/releasedocmake EOF chmod +x "${bin_tarball}/bin/releasedocmaker" -for utility in shelldocs/shelldocs.py \ +for utility in shelldocs/shelldocs.py precommit/qbt.sh \ precommit/smart-apply-patch.sh precommit/test-patch.sh do wrapper=${utility##*/} http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/core.d/00-yetuslib.sh ---------------------------------------------------------------------- diff --git a/precommit/core.d/00-yetuslib.sh b/precommit/core.d/00-yetuslib.sh index 90ffa16..25118cb 100755 --- a/precommit/core.d/00-yetuslib.sh +++ b/precommit/core.d/00-yetuslib.sh @@ -101,7 +101,7 @@ function yetus_run_and_redirect ## @description Given a filename or dir, return the absolute version of it ## @audience public ## @stability stable -## @param directory +## @param fsobj ## @replaceable no ## @return 0 success ## @return 1 failure http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/core.d/01-common.sh ---------------------------------------------------------------------- diff --git a/precommit/core.d/01-common.sh b/precommit/core.d/01-common.sh index 0661f6e..f31d613 100755 --- a/precommit/core.d/01-common.sh +++ b/precommit/core.d/01-common.sh @@ -536,6 +536,9 @@ function verify_command yetus_error "executable for '${cmd_name}' was not specified." return 1 fi + if [[ ! "${cmd_path}" =~ / ]]; then + cmd_path=$(command -v "${cmd_path}") + fi if [[ ! -f ${cmd_path} ]]; then yetus_error "executable '${cmd_path}' for '${cmd_name}' does not exist." return 1 @@ -546,3 +549,22 @@ function verify_command fi return 0 } + +## @description Faster dirname, given the assumption that +## @description dirs are always absolute (e.g., start with /) +## @description DO NOT USE with relative paths or where +## @description assumption may not be valid! +## @audience private +## @stability evolving +## @replaceable no +## @param fileobj +function faster_dirname +{ + declare o=$1 + + if [[ "${o}" =~ / ]]; then + echo "${o%/*}" + else + echo . + fi +} http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/core.d/builtin-bugsystem.sh ---------------------------------------------------------------------- diff --git a/precommit/core.d/builtin-bugsystem.sh b/precommit/core.d/builtin-bugsystem.sh index f83e7c9..c7d9a6c 100755 --- a/precommit/core.d/builtin-bugsystem.sh +++ b/precommit/core.d/builtin-bugsystem.sh @@ -45,6 +45,11 @@ function console_finalreport declare seccoladj=0 declare spcfx=${PATCH_DIR}/spcl.txt + if [[ -n "${CONSOLE_REPORT_FILE}" ]]; then + exec 6>&1 + exec >"${CONSOLE_REPORT_FILE}" + fi + if [[ ${result} == 0 ]]; then if [[ ${ROBOT} == false ]]; then if declare -f ${PROJECT_NAME}_console_success >/dev/null; then @@ -154,4 +159,9 @@ function console_finalreport printf "%s\n" "${comment}" ((i=i+1)) done + + if [[ -n "${CONSOLE_REPORT_FILE}" ]]; then + exec 1>&6 6>&- + cat "${CONSOLE_REPORT_FILE}" + fi } http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/core.d/docker.sh ---------------------------------------------------------------------- diff --git a/precommit/core.d/docker.sh b/precommit/core.d/docker.sh index e0706a3..062c8c1 100755 --- a/precommit/core.d/docker.sh +++ b/precommit/core.d/docker.sh @@ -23,6 +23,8 @@ DOCKERFAIL="fallback,continue,fail" DOCKERSUPPORT=false DOCKER_ENABLE_PRIVILEGED=true +declare -a DOCKER_EXTRAARGS + #### #### IMPORTANT #### @@ -531,7 +533,7 @@ function docker_run_image cleanup_and_exit 1 fi - big_console_header "Building patch image: ${patchimagename}" + big_console_header "Building ${BUILDMODE} image: ${patchimagename}" start_clock # using the base image, make one that is patch specific dockercmd build \ @@ -564,14 +566,13 @@ PatchSpecificDocker cleanup_and_exit 1 fi - if [[ -f "${PATCH_DIR}/buildtool-docker-params.txt" ]]; then - extraargs=$(cat "${PATCH_DIR}/buildtool-docker-params.txt") - else - extraargs="" + if [[ "${DOCKER_ENABLE_PRIVILEGED}" = true ]]; then + DOCKER_EXTRAARGS=("--privileged" "${DOCKER_EXTRAARGS[@]}") fi - if [[ "${DOCKER_ENABLE_PRIVILEGED}" = true ]]; then - extraargs="${extraargs} --privileged " + if [[ -n "${CONSOLE_REPORT_FILE}" ]]; then + touch "${CONSOLE_REPORT_FILE}" + DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${CONSOLE_REPORT_FILE}:/testptch/console.txt") fi client=$(docker_version Client) @@ -579,9 +580,8 @@ PatchSpecificDocker dockerversion="Client=${client} Server=${server}" if [[ ${PATCH_DIR} =~ ^/ ]]; then - # shellcheck disable=SC2086 exec "${DOCKERCMD}" run --rm=true -i \ - ${extraargs} \ + "${DOCKER_EXTRAARGS[@]}" \ -v "${PWD}:/testptch/${PROJECT_NAME}" \ -v "${PATCH_DIR}:/testptch/patchprocess" \ -u "${USER_NAME}" \ @@ -596,9 +596,8 @@ PatchSpecificDocker --name "${containername}" \ "${patchimagename}" else - # shellcheck disable=SC2086 exec "${DOCKERCMD}" run --rm=true -i \ - ${extraargs} \ + "${DOCKER_EXTRAARGS[@]}" \ -v "${PWD}:/testptch/${PROJECT_NAME}" \ -u "${USER_NAME}" \ -w "/testptch/${PROJECT_NAME}" \ @@ -612,6 +611,9 @@ PatchSpecificDocker --name "${containername}" \ "${patchimagename}" fi + + # this should never get reached, but we put it here just in case + cleanup_and_exit 1 } ## @description Switch over to a Docker container http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/personality/flink.sh ---------------------------------------------------------------------- diff --git a/precommit/personality/flink.sh b/precommit/personality/flink.sh index 6b4f0a8..76fa418 100755 --- a/precommit/personality/flink.sh +++ b/precommit/personality/flink.sh @@ -31,6 +31,10 @@ function personality_globals add_test_type flinklib +## @description flinklib file filter +## @audience private +## @stability evolving +## @param filename function flinklib_filefilter { local filename=$1 @@ -42,6 +46,9 @@ function flinklib_filefilter fi } +## @description count files in the lib dir +## @audience private +## @stability evolving function flinklib_count { find "${BASEDIR}" \ @@ -50,42 +57,46 @@ function flinklib_count | wc -l } -function flinklib_preapply +## @description check fliblib +## @audience private +## @stability evolving +## @param repostatus +function flinklib_rebuild { + declare repostatus=$1 + start_clock big_console_header "${PATCH_BRANCH} flink library dependencies" if ! verify_needed_test flinklib; then - echo "Patch does not need flinklib testing." + echo "${BUILDMODEMSG} does not need flinklib testing." return 0 fi pushd "${BASEDIR}" >/dev/null - echo_and_redirect "${PATCH_DIR}/branch-flinklib-root.txt" \ + echo_and_redirect "${PATCH_DIR}/${repostatus}-flinklib-root.txt" \ "${MAVEN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch if [[ $? != 0 ]]; then add_vote_table -1 flinklib "Unable to determine flink libs in ${PATCH_BRANCH}." fi - FLINK_PRE_LIB_FILES=$(flinklib_count) - popd >/dev/null -} -function flinklib_postapply -{ - start_clock - big_console_header "Patch flink library dependencies" - - if ! verify_needed_test flinklib; then - echo "Patch does not need flinklib testing." + if [[ ${repostatus} = branch ]]; then + FLINK_PRE_LIB_FILES=$(flinklib_count) return 0 + else + FLINK_POST_LIB_FILES=$(flinklib_count) fi - - pushd "${BASEDIR}" >/dev/null - echo_and_redirect "${PATCH_DIR}/patch-flinklib-root.txt" \ - "${MAVEN}" "${MAVEN_ARGS[@]}" package -DskipTests -Dmaven.javadoc.skip=true -Ptest-patch - FLINK_POST_LIB_FILES=$(flinklib_count) popd >/dev/null + if [[ "${BUILDMODE}" = full ]]; then + if [[ ${FLINK_POST_LIB_FILES} -gt 0 ]]; then + add_vote_table -1 flinklib "Lib folder dependencies are currently ${FLINK_POST_LIB_FILES}" + return 1 + else + add_vote_table +1 flinklib "No lib folder dependencies!" + return 0 + fi + fi if [[ "${FLINK_POST_LIB_FILES}" -gt "${FLINK_PRE_LIB_FILES}" ]]; then add_vote_table -1 flinklib "Patch increases lib folder dependencies from " \ @@ -99,16 +110,4 @@ function flinklib_postapply "$((FLINK_PRE_LIB_FILES-FLINK_POST_LIB_FILES))." fi return 0 -} - -function flinklib_rebuild -{ - declare repostatus=$1 - - if [[ "${repostatus}" = branch ]]; then - flinklib_preapply - else - flinklib_postinstall - fi -} - +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/personality/hadoop.sh ---------------------------------------------------------------------- diff --git a/precommit/personality/hadoop.sh b/precommit/personality/hadoop.sh index 59dfd2c..5b96994 100755 --- a/precommit/personality/hadoop.sh +++ b/precommit/personality/hadoop.sh @@ -46,8 +46,12 @@ function hadoop_order if [[ ${ordering} = normal ]]; then hadoopm="${CHANGED_MODULES[*]}" - elif [[ ${ordering} = union ]]; then + elif [[ ${ordering} = union ]]; then hadoopm="${CHANGED_UNION_MODULES}" + elif [[ ${ordering} = mvnsrc ]]; then + hadoopm="${MAVEN_SRC_MODULES[*]}" + elif [[ ${ordering} = mvnsrctest ]]; then + hadoopm="${MAVEN_SRCTEST_MODULES[*]}" else hadoopm="${ordering}" fi @@ -205,7 +209,7 @@ function personality_modules ordering=. fi - if [[ ${repostatus} = patch ]]; then + if [[ "${repostatus}" = patch && "${BUILDMODE}" = patch ]]; then echo "javadoc pre-reqs:" for i in hadoop-project \ hadoop-common-project/hadoop-annotations; do @@ -226,7 +230,7 @@ function personality_modules ;; mvninstall) extra="-DskipTests" - if [[ ${repostatus} = branch ]]; then + if [[ "${repostatus}" = branch || "${BUILDMODE}" = full ]]; then ordering=. fi ;; @@ -236,7 +240,9 @@ function personality_modules fi ;; unit) - if [[ "${CHANGED_MODULES[*]}" =~ \. ]]; then + if [[ "${BUILDMODE}" = full ]]; then + ordering=mvnsrc + elif [[ "${CHANGED_MODULES[*]}" =~ \. ]]; then ordering=. fi http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/personality/hbase.sh ---------------------------------------------------------------------- diff --git a/precommit/personality/hbase.sh b/precommit/personality/hbase.sh index 352d2e3..5e6f0dc 100755 --- a/precommit/personality/hbase.sh +++ b/precommit/personality/hbase.sh @@ -22,17 +22,23 @@ personality_plugins "all" function personality_globals { #shellcheck disable=SC2034 + PROJECT_NAME=hbase + #shellcheck disable=SC2034 PATCH_BRANCH_DEFAULT=master #shellcheck disable=SC2034 JIRA_ISSUE_RE='^HBASE-[0-9]+$' #shellcheck disable=SC2034 GITHUB_REPO="apache/hbase" + # TODO use PATCH_BRANCH to select hadoop versions to use. # All supported Hadoop versions that we want to test the compilation with - HBASE_HADOOP_VERSIONS="2.4.1 2.5.2 2.6.0" + HBASE_HADOOP_VERSIONS="2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1" + + # TODO use PATCH_BRANCH to select jdk versions to use. # Override the maven options MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx3100M"}" + } ## @description Queue up modules for this personality @@ -53,16 +59,20 @@ function personality_modules extra="-DHBasePatchProcess" if [[ ${repostatus} == branch - && ${testtype} == mvninstall ]];then - personality_enqueue_module . ${extra} - return - fi + && ${testtype} == mvninstall ]] || + [[ "${BUILDMODE}" == full ]];then + personality_enqueue_module . ${extra} + return + fi if [[ ${testtype} = findbugs ]]; then for module in "${CHANGED_MODULES[@]}"; do - # skip findbugs on hbase-shell + # skip findbugs on hbase-shell and hbase-it. hbase-it has nothing + # in src/main/java where findbugs goes to look if [[ ${module} == hbase-shell ]]; then continue + elif [[ ${module} == hbase-it ]]; then + continue else # shellcheck disable=SC2086 personality_enqueue_module ${module} ${extra} @@ -71,6 +81,16 @@ function personality_modules return fi + if [[ ${testtype} = unit ]]; then + extra="${extra} -PrunAllTests" + + # Inject the jenkins build-id for our surefire invocations + # Used by zombie detection stuff, even though we're not including that yet. + if [ -n "${BUILD_ID}" ]; then + extra="${extra} -Dbuild.id=${BUILD_ID}" + fi + fi + for module in "${CHANGED_MODULES[@]}"; do # shellcheck disable=SC2086 personality_enqueue_module ${module} ${extra} @@ -78,9 +98,19 @@ function personality_modules } ################################################### +# Below here are our one-off tests specific to hbase. +# TODO break them into individual files so it's easier to maintain them? + +# TODO line length check? could ignore all java files since checkstyle gets them. + +################################################### add_test_type hadoopcheck +## @description hadoopcheck file filter +## @audience private +## @stability evolving +## @param filename function hadoopcheck_filefilter { local filename=$1 @@ -90,6 +120,10 @@ function hadoopcheck_filefilter fi } +## @description hadoopcheck test +## @audience private +## @stability evolving +## @param repostatus function hadoopcheck_rebuild { local repostatus=$1 @@ -113,7 +147,7 @@ function hadoopcheck_rebuild -Dhadoop-two.version="${hadoopver}" count=$(${GREP} -c ERROR "${logfile}") if [[ ${count} -gt 0 ]]; then - add_vote_table -1 hadoopcheck "Patch causes ${count} errors with Hadoop v${hadoopver}." + add_vote_table -1 hadoopcheck "${BUILDMODEMSG} causes ${count} errors with Hadoop v${hadoopver}." ((result=result+1)) fi done @@ -122,14 +156,19 @@ function hadoopcheck_rebuild return 1 fi - add_vote_table +1 hadoopcheck "Patch does not cause any errors with Hadoop ${HBASE_HADOOP_VERSIONS}." + add_vote_table +1 hadoopcheck "${BUILDMODEMSG} does not cause any errors with Hadoop ${HBASE_HADOOP_VERSIONS}." return 0 } ###################################### +# TODO if we need the protoc check, we probably need to check building all the modules that rely on hbase-protocol add_test_type hbaseprotoc +## @description hbaseprotoc file filter +## @audience private +## @stability evolving +## @param filename function hbaseprotoc_filefilter { local filename=$1 @@ -139,33 +178,38 @@ function hbaseprotoc_filefilter fi } +## @description hadoopcheck test +## @audience private +## @stability evolving +## @param repostatus function hbaseprotoc_rebuild { - local i=0 - local fn - local module - local logfile - local count - local result + declare repostatus=$1 + declare i=0 + declare fn + declare module + declare logfile + declare count + declare result if [[ "${repostatus}" = branch ]]; then return 0 fi - if ! verify_needed_test hbaseprotoc; then + verify_needed_test hbaseprotoc + if [[ $? == 0 ]]; then return 0 fi - big_console_header "Patch HBase protoc plugin" + big_console_header "HBase protoc plugin: ${BUILDMODE}" start_clock - personality_modules patch hbaseprotoc modules_workers patch hbaseprotoc compile -DskipTests -Pcompile-protobuf -X -DHBasePatchProcess # shellcheck disable=SC2153 - until [[ $i -eq ${#MODULE[@]} ]]; do + until [[ $i -eq "${#MODULE[@]}" ]]; do if [[ ${MODULE_STATUS[${i}]} == -1 ]]; then ((result=result+1)) ((i=i+1)) @@ -196,6 +240,10 @@ function hbaseprotoc_rebuild add_test_type hbaseanti +## @description hbaseanti file filter +## @audience private +## @stability evolving +## @param filename function hbaseanti_filefilter { local filename=$1 @@ -205,13 +253,22 @@ function hbaseanti_filefilter fi } +## @description hbaseanti patch file check +## @audience private +## @stability evolving +## @param filename function hbaseanti_patchfile { local patchfile=$1 local warnings local result - if ! verify_needed_test hbaseanti; then + if [[ "${BUILDMODE}" = full ]]; then + return 0 + fi + + verify_needed_test hbaseanti + if [[ $? == 0 ]]; then return 0 fi @@ -238,3 +295,75 @@ function hbaseanti_patchfile add_vote_table +1 hbaseanti "" "Patch does not have any anti-patterns." return 0 } + + +## @description hbase custom mvnsite file filter. See HBASE-15042 +## @audience private +## @stability evolving +## @param filename +function mvnsite_filefilter +{ + local filename=$1 + + if [[ ${BUILDTOOL} = maven ]]; then + if [[ ${filename} =~ src/main/site || ${filename} =~ src/main/asciidoc ]]; then + yetus_debug "tests/mvnsite: ${filename}" + add_test mvnsite + fi + fi +} + +## This is named so that yetus will check us right after running tests. +## Essentially, we check for normal failures and then we look for zombies. +#function hbase_unit_logfilter +#{ +# declare testtype="unit" +# declare input=$1 +# declare output=$2 +# declare processes +# declare process_output +# declare zombies +# declare zombie_count=0 +# declare zombie_process +# +# yetus_debug "in hbase-specific unit logfilter." +# +# # pass-through to whatever is counting actual failures +# if declare -f ${BUILDTOOL}_${testtype}_logfilter >/dev/null; then +# "${BUILDTOOL}_${testtype}_logfilter" "${input}" "${output}" +# elif declare -f ${testtype}_logfilter >/dev/null; then +# "${testtype}_logfilter" "${input}" "${output}" +# fi +# +# start_clock +# if [ -n "${BUILD_ID}" ]; then +# yetus_debug "Checking for zombie test processes." +# processes=$(jps -v | "${GREP}" surefirebooter | "${GREP}" -e "hbase.build.id=${BUILD_ID}") +# if [ -n "${processes}" ] && [ "$(echo "${processes}" | wc -l)" -gt 0 ]; then +# yetus_warn "Found some suspicious process(es). Waiting a bit to see if they're just slow to stop." +# yetus_debug "${processes}" +# sleep 30 +# #shellcheck disable=SC2016 +# for pid in $(echo "${processes}"| ${AWK} '{print $1}'); do +# # Test our zombie still running (and that it still an hbase build item) +# process_output=$(ps -p "${pid}" | tail +2 | "${GREP}" -e "hbase.build.id=${BUILD_ID}") +# if [[ -n "${process_output}" ]]; then +# yetus_error "Zombie: ${process_output}" +# ((zombie_count = zombie_count + 1)) +# zombie_process=$(jstack "${pid}" | "${GREP}" -e "\.Test" | "${GREP}" -e "\.java"| head -3) +# zombies="${zombies} ${zombie_process}" +# fi +# done +# fi +# if [ "${zombie_count}" -ne 0 ]; then +# add_vote_table -1 zombies "There are ${zombie_count} zombie test(s)" +# populate_test_table "zombie unit tests" "${zombies}" +# else +# yetus_info "Zombie check complete. All test runs exited normally." +# stop_clock +# fi +# else +# add_vote_table -0 zombies "There is no BUILD_ID env variable; can't check for zombies." +# fi +# +#} http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/personality/jmeter.sh ---------------------------------------------------------------------- diff --git a/precommit/personality/jmeter.sh b/precommit/personality/jmeter.sh index 194fc4f..b4e61b5 100755 --- a/precommit/personality/jmeter.sh +++ b/precommit/personality/jmeter.sh @@ -58,6 +58,9 @@ function jmeter_parse_args done } +## @description Download jmetere dependencies +## @audience private +## @stability evolving function jmeter_precheck { if [[ ${JMETER_DOWNLOAD_JARS} = true ]]; then http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch-docker/launch-test-patch.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch-docker/launch-test-patch.sh b/precommit/test-patch-docker/launch-test-patch.sh index 7daa612..f15dbf8 100755 --- a/precommit/test-patch-docker/launch-test-patch.sh +++ b/precommit/test-patch-docker/launch-test-patch.sh @@ -14,6 +14,12 @@ # See the License for the specific language governing permissions and # limitations under the License. +declare -a OVERWRITEARGS + +OVERWRITEARGS=("--reexec") +OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--dockermode") +OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--basedir=${BASEDIR}") + cd "${BASEDIR}" || exit 1 if [[ -n ${JAVA_HOME} @@ -25,6 +31,10 @@ fi if [[ -z ${JAVA_HOME} ]]; then JAVA_HOME=$(find /usr/lib/jvm/ -name "java-*" -type d | tail -1) export JAVA_HOME + if [[ -n "${JAVA_HOME}" ]]; then + OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--java-home=${JAVA_HOME}") + echo "Setting ${JAVA_HOME} as the JAVA_HOME." + fi fi # Avoid out of memory errors in builds @@ -35,9 +45,9 @@ export MAVEN_OPTS TESTPATCHMODE=${TESTPATCHMODE/--docker } TESTPATCHMODE=${TESTPATCHMODE%--docker} - -cd "${BASEDIR}" || exit 1 PATCH_DIR=$(cd -P -- "${PATCH_DIR}" >/dev/null && pwd -P) +OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--patch-dir=${PATCH_DIR}") +OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--user-plugins=${PATCH_DIR}/precommit/user-plugins") # if patch system is generic, then it's either a local # patch file or was in some other way not pulled from a bug @@ -45,16 +55,15 @@ PATCH_DIR=$(cd -P -- "${PATCH_DIR}" >/dev/null && pwd -P) # test-patch where to find it. if [[ "${PATCH_SYSTEM}" = generic ]]; then cp -p "${PATCH_DIR}/patch" /testptch/extras/patch - patchfile="/testptch/extras/patch" + OVERWRITEARGS=("${OVERWRITEARGS[@]}" "/testptch/extras/patch") +fi + +if [[ -f /testptch/console.txt ]]; then + OVERWRITEARGS=("${OVERWRITEARGS[@]}" "--console-report-file=/testptch/console.txt") fi cd "${PATCH_DIR}/precommit/" || exit 1 #shellcheck disable=SC2086 "${PATCH_DIR}/precommit/test-patch.sh" \ - --reexec \ - --dockermode ${TESTPATCHMODE} \ - --basedir="${BASEDIR}" \ - --patch-dir="${PATCH_DIR}" \ - --java-home="${JAVA_HOME}" \ - --user-plugins="${PATCH_DIR}/precommit/user-plugins" \ - ${patchfile} + ${TESTPATCHMODE} \ + "${OVERWRITEARGS[@]}" http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/ant.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/ant.sh b/precommit/test-patch.d/ant.sh index 64c4067..61e13c9 100755 --- a/precommit/test-patch.d/ant.sh +++ b/precommit/test-patch.d/ant.sh @@ -200,5 +200,5 @@ function ant_builtin_personality_file_tests function ant_docker_support { - echo "-v ${HOME}/.ivy2:${HOME}/.ivy2" > "${PATCH_DIR}/buildtool-docker-params.txt" + DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS}" "-v" "${HOME}/.ivy2:${HOME}/.ivy2") } http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/asflicense.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/asflicense.sh b/precommit/test-patch.d/asflicense.sh index edf7a7f..593e4af 100755 --- a/precommit/test-patch.d/asflicense.sh +++ b/precommit/test-patch.d/asflicense.sh @@ -61,7 +61,7 @@ function asflicense_tests local numpatch local btfails=true - big_console_header "Determining number of patched ASF License errors" + big_console_header "Determining number of ASF License errors" start_clock @@ -105,7 +105,7 @@ function asflicense_tests # RAT fails the build if there are license problems. # so let's take advantage of that a bit. if [[ $? == 0 && ${btfails} = true ]]; then - add_vote_table 1 asflicense "Patch does not generate ASF License warnings." + add_vote_table 1 asflicense "${BUILDMODEMSG} does not generate ASF License warnings." return 0 fi @@ -134,7 +134,7 @@ function asflicense_tests echo "There appear to be ${numpatch} ASF License warnings after applying the patch." if [[ -n ${numpatch} && ${numpatch} -gt 0 ]] ; then - add_vote_table -1 asflicense "Patch generated ${numpatch} ASF License warnings." + add_vote_table -1 asflicense "${BUILDMODEMSG} generated ${numpatch} ASF License warnings." echo "Lines that start with ????? in the ASF License "\ "report indicate files that do not have an Apache license header:" \ @@ -146,7 +146,7 @@ function asflicense_tests add_footer_table asflicense "@@BASE@@/patch-asflicense-problems.txt" return 1 fi - add_vote_table 1 asflicense "Patch does not generate ASF License warnings." + add_vote_table 1 asflicense "${BUILDMODEMSG} does not generate ASF License warnings." return 0 } http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/author.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/author.sh b/precommit/test-patch.d/author.sh index 4497d03..dccd504 100755 --- a/precommit/test-patch.d/author.sh +++ b/precommit/test-patch.d/author.sh @@ -16,21 +16,83 @@ add_test_type author -## @description Check the current directory for @author tags +## @description author usage hook +## @audience private +## @stability evolving +## @replaceable no +function author_usage +{ + yetus_add_option "--author-ignore-list=" "list of filenames to ignore (full build mode only)" +} + +## @description author parse args hook +## @audience private +## @stability evolving +## @replaceable no +function author_parse_args +{ + declare i + + for i in "$@"; do + case ${i} in + --author-ignore-list=*) + yetus_comma_to_array AUTHOR_IGNORE_LIST "${i#*=}" + ;; + esac + done +} + +## @description helper function for @author tags check ## @audience private ## @stability evolving ## @replaceable no ## @return 0 on success ## @return 1 on failure +function author_generic +{ + declare authortags + declare i + declare msg + + if [[ "${BUILDMODE}" = full ]]; then + msg="source tree" + else + msg="patch" + fi + + # shellcheck disable=SC2016 + authortags=$(wc -l "${PATCH_DIR}/author-tags.txt" | "${AWK}" '{print $1}') + echo "There appear to be ${authortags} @author tags in the ${msg}." + if [[ ${authortags} != 0 ]] ; then + add_vote_table -1 @author \ + "${BUILDMODEMSG} appears to contain ${authortags} @author tags which the" \ + " community has agreed to not allow in code contributions." + add_footer_table @author "@@BASE@@/author-tags.txt" + return 1 + fi + add_vote_table +1 @author "${BUILDMODEMSG} does not contain any @author tags." + return 0 +} + +## @description Check the current patchfile for @author tags +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +## @param patchfile function author_patchfile { declare patchfile=$1 - declare authorTags # shellcheck disable=SC2155 declare -r appname=$(basename "${BASH_SOURCE-$0}") declare i - big_console_header "Checking there are no @author tags in the patch." + if [[ "${BUILDMODE}" != patch ]]; then + return + fi + + big_console_header "Checking for @author tags: ${BUILDMODE}" start_clock @@ -43,16 +105,44 @@ function author_patchfile done ${GREP} -i -n '^[^-].*@author' "${patchfile}" >> "${PATCH_DIR}/author-tags.txt" - # shellcheck disable=SC2016 - authorTags=$(wc -l "${PATCH_DIR}/author-tags.txt" | "${AWK}" '{print $1}') - echo "There appear to be ${authorTags} @author tags in the patch." - if [[ ${authorTags} != 0 ]] ; then - add_vote_table -1 @author \ - "The patch appears to contain ${authorTags} @author tags which the" \ - " community has agreed to not allow in code contributions." - add_footer_table @author "@@BASE@@/author-tags.txt" - return 1 + author_generic +} + + +## @description Check the current directory for @author tags +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure +function author_postcompile +{ + # shellcheck disable=SC2155 + declare -r appname=$(basename "${BASH_SOURCE-$0}") + declare fn + + if [[ "${BUILDMODE}" != full ]]; then + return fi - add_vote_table +1 @author "The patch does not contain any @author tags." - return 0 + + big_console_header "Checking for @author tags: ${BUILDMODE}" + + start_clock + + "${GIT}" grep -n -I --extended-regexp -i -e '^[^-].*@author' \ + | ${GREP} -v "${appname}" \ + >> "${PATCH_DIR}/author-tags-git.txt" + + if [[ -z "${AUTHOR_IGNORE_LIST[0]}" ]]; then + cp -p "${PATCH_DIR}/author-tags-git.txt" "${PATCH_DIR}/author-tags.txt" + else + cp -p "${PATCH_DIR}/author-tags-git.txt" "${PATCH_DIR}/author-tags.1" + for fn in "${AUTHOR_IGNORE_LIST[@]}"; do + ${GREP} -v -E "^${fn}" "${PATCH_DIR}/author-tags.1" >> "${PATCH_DIR}/author-tags.2" + mv "${PATCH_DIR}/author-tags.2" "${PATCH_DIR}/author-tags.1" + done + mv "${PATCH_DIR}/author-tags.1" "${PATCH_DIR}/author-tags.txt" + fi + + author_generic } http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/autoconf.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/autoconf.sh b/precommit/test-patch.d/autoconf.sh index 2937e72..933a8c0 100755 --- a/precommit/test-patch.d/autoconf.sh +++ b/precommit/test-patch.d/autoconf.sh @@ -91,9 +91,9 @@ function autoconf_precompile fi if [[ "${repostatus}" = branch ]]; then - big_console_header "Pre-patch ${PATCH_BRANCH} autoconf verification" + big_console_header "autoconf verification: ${PATCH_BRANCH}" else - big_console_header "Patch autoconf verification" + big_console_header "autoconf verification: ${BUILDMODE}" fi personality_modules "${repostatus}" autoreconf @@ -108,7 +108,7 @@ function autoconf_precompile # shellcheck disable=SC2153 add_vote_table -1 autoreconf "${PATCH_BRANCH} unable to autoreconf" else - add_vote_table -1 autoreconf "${repostatus} unable to autoreconf" + add_vote_table -1 autoreconf "${BUILDMODEMSG} is unable to autoreconf" fi add_footer_table "autoreconf" "@@BASE@@/${repostatus}-autoconf-autoreconf" return 1 @@ -117,7 +117,7 @@ function autoconf_precompile # shellcheck disable=SC2153 add_vote_table +1 autoreconf "${PATCH_BRANCH} autoreconf successful" else - add_vote_table +1 autoreconf "${repostatus} autoreconf successful" + add_vote_table +1 autoreconf "${BUILDMODEMSG} can autoreconf" fi fi @@ -138,7 +138,7 @@ function autoconf_precompile # shellcheck disable=SC2153 add_vote_table -1 configure "${PATCH_BRANCH} unable to configure" else - add_vote_table -1 configure "${repostatus} unable to configure" + add_vote_table -1 configure "${BUILDMODEMSG} is unable to configure" fi add_footer_table "configure" "@@BASE@@/${repostatus}-autoconf-configure" return 1 @@ -147,7 +147,7 @@ function autoconf_precompile # shellcheck disable=SC2153 add_vote_table +1 configure "${PATCH_BRANCH} configure successful" else - add_vote_table +1 configure "${repostatus} configure successful" + add_vote_table +1 configure "${BUILDMODEMSG} can configure" fi fi return 0 @@ -201,4 +201,4 @@ function autoconf_builtin_personality_file_tests else make_builtin_personality_file_tests "${filename}" fi -} \ No newline at end of file +} http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/checkstyle.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/checkstyle.sh b/precommit/test-patch.d/checkstyle.sh index 4363e4d..f0b3e57 100755 --- a/precommit/test-patch.d/checkstyle.sh +++ b/precommit/test-patch.d/checkstyle.sh @@ -140,7 +140,6 @@ function checkstyle_runner declare savestop declare output declare logfile - declare repo declare modulesuffix declare cmd declare logline @@ -151,12 +150,6 @@ function checkstyle_runner # first, let's clear out any previous run information modules_reset - if [[ ${repostatus} == branch ]]; then - repo=${PATCH_BRANCH} - else - repo="the patch" - fi - # loop through the modules we've been given #shellcheck disable=SC2153 until [[ $i -eq ${#MODULE[@]} ]]; do @@ -200,14 +193,14 @@ function checkstyle_runner > "${tmp}" if [[ $? == 0 ]] ; then - module_status ${i} +1 "${logfile}" "${modulesuffix} in ${repo} passed checkstyle" + module_status ${i} +1 "${logfile}" "${BUILDMODEMSG} ${modulesuffix} passed checkstyle" else - module_status ${i} -1 "${logfile}" "${modulesuffix} in ${repo} failed checkstyle" + module_status ${i} -1 "${logfile}" "${BUILDMODEMSG} ${modulesuffix} failed checkstyle" ((result = result + 1)) fi # if we have some output, we need to do more work: - if [[ -s ${tmp} ]]; then + if [[ -s ${tmp} && "${BUILDMODE}" = patch ]]; then # first, let's pull out all of the files that # we actually care about, esp since that run @@ -219,6 +212,7 @@ function checkstyle_runner ${GREP} "${j}" "${tmp}" >> "${tmp}.1" done + # now that we have just the files we care about, # let's unscrew it. You see... @@ -248,7 +242,8 @@ function checkstyle_runner popd >/dev/null # later on, calcdiff will turn this into code(:column):error # compare, and then put the file:line back onto it. - + else + cp -p "${tmp}" "${output}" fi rm "${tmp}" "${tmp}.1" 2>/dev/null @@ -288,7 +283,7 @@ function checkstyle_preapply return 0 fi - big_console_header "${PATCH_BRANCH} checkstyle" + big_console_header "checkstyle: ${PATCH_BRANCH}" start_clock @@ -321,7 +316,7 @@ function checkstyle_postapply return 0 fi - big_console_header "Patch checkstyle plugin" + big_console_header "checkstyle: ${BUILDMODE}" start_clock @@ -346,12 +341,21 @@ function checkstyle_postapply module=${MODULE[$i]} fn=$(module_file_fragment "${module}") - # call calcdiffs to allow overrides - calcdiffs \ - "${PATCH_DIR}/branch-checkstyle-${fn}.txt" \ - "${PATCH_DIR}/patch-checkstyle-${fn}.txt" \ - checkstyle \ - > "${PATCH_DIR}/diff-checkstyle-${fn}.txt" + # if there is no comparison to be done, + # we can speed this up tremendously + if [[ "${BUILDMODE}" = full ]]; then + touch "${PATCH_DIR}/branch-checkstyle-${fn}.txt" + cp -p "${PATCH_DIR}/patch-checkstyle-${fn}.txt" \ + "${PATCH_DIR}/diff-checkstyle-${fn}.txt" + else + + # call calcdiffs to allow overrides + calcdiffs \ + "${PATCH_DIR}/branch-checkstyle-${fn}.txt" \ + "${PATCH_DIR}/patch-checkstyle-${fn}.txt" \ + checkstyle \ + > "${PATCH_DIR}/diff-checkstyle-${fn}.txt" + fi #shellcheck disable=SC2016 numbranch=$(wc -l "${PATCH_DIR}/branch-checkstyle-${fn}.txt" | ${AWK} '{print $1}') @@ -371,9 +375,9 @@ function checkstyle_postapply if [[ ${addpatch} -gt 0 ]] ; then ((result = result + 1)) - module_status ${i} -1 "diff-checkstyle-${fn}.txt" "${mod}: patch ${statstring}" + module_status ${i} -1 "diff-checkstyle-${fn}.txt" "${mod}: ${BUILDMODEMSG} ${statstring}" elif [[ ${fixedpatch} -gt 0 ]]; then - module_status ${i} +1 "diff-checkstyle-${fn}.txt" "${mod}: patch ${statstring}" + module_status ${i} +1 "diff-checkstyle-${fn}.txt" "${mod}: ${BUILDMODEMSG} ${statstring}" summarize=false fi ((i=i+1)) http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/cmake.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/cmake.sh b/precommit/test-patch.d/cmake.sh index 9b5f265..7e68a4d 100755 --- a/precommit/test-patch.d/cmake.sh +++ b/precommit/test-patch.d/cmake.sh @@ -128,9 +128,9 @@ function cmake_precompile if [[ "${repostatus}" = branch ]]; then # shellcheck disable=SC2153 - big_console_header "${PATCH_BRANCH} cmake CMakeLists.txt" + big_console_header "cmake CMakeLists.txt: ${PATCH_BRANCH}" else - big_console_header "Patch cmake CMakeLists.txt" + big_console_header "cmake CMakeLists.txt: ${BUILDMODE}" fi personality_modules "${repostatus}" CMakeLists.txt http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/findbugs.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/findbugs.sh b/precommit/test-patch.d/findbugs.sh index e4399fc..24ccc95 100755 --- a/precommit/test-patch.d/findbugs.sh +++ b/precommit/test-patch.d/findbugs.sh @@ -95,10 +95,8 @@ function findbugs_precheck ## @audience private ## @stability evolving ## @replaceable no -## @param repostatus function findbugs_maven_skipper { - declare repostat=$1 declare -i i=0 declare skiplist=() declare modname @@ -122,7 +120,11 @@ function findbugs_maven_skipper done if [[ -n "${modname}" ]]; then - add_vote_table 0 findbugs "Skipped ${repostat} modules with no Java source: ${skiplist[*]}" + if [[ "${BUILDMODE}" = patch ]]; then + add_vote_table 0 findbugs "Skipped patched modules with no Java source: ${skiplist[*]}" + else + add_vote_table 0 findbugs "Skipped ${#skiplist[@]} modules in the source tree with no Java source." + fi fi } @@ -132,6 +134,7 @@ function findbugs_maven_skipper ## @replaceable no ## @return 0 on success ## @return 1 on failure +## @param repostatus function findbugs_runner { local name=$1 @@ -147,7 +150,7 @@ function findbugs_runner # strip out any modules that aren't actually java modules # this can save a lot of time during testing if [[ "${BUILDTOOL}" = maven ]]; then - findbugs_maven_skipper "${name}" + findbugs_maven_skipper fi "${BUILDTOOL}_modules_worker" "${name}" findbugs @@ -248,7 +251,7 @@ function findbugs_preapply return 0 fi - big_console_header "Pre-patch findbugs detection" + big_console_header "findbugs detection: ${PATCH_BRANCH}" findbugs_runner branch result=$? @@ -278,9 +281,19 @@ function findbugs_preapply if [[ ${module_findbugs_warnings} -gt 0 ]] ; then msg="${module} in ${PATCH_BRANCH} has ${module_findbugs_warnings} extant Findbugs warnings." - if [[ "${FINDBUGS_WARNINGS_FAIL_PRECHECK}" == "true" ]]; then + if [[ "${FINDBUGS_WARNINGS_FAIL_PRECHECK}" = "true" ]]; then + module_status ${modindex} -1 "branch-findbugs-${fn}-warnings.html" "${msg}" + ((result=result+1)) + elif [[ "${BUILDMODE}" = full ]]; then module_status ${modindex} -1 "branch-findbugs-${fn}-warnings.html" "${msg}" ((result=result+1)) + populate_test_table FindBugs "module:${module}" + #shellcheck disable=SC2162 + while read line; do + firstpart=$(echo "${line}" | cut -f2 -d:) + secondpart=$(echo "${line}" | cut -f9- -d' ') + add_test_table "" "${firstpart}:${secondpart}" + done < <("${FINDBUGS_HOME}/bin/convertXmlToText" "${warnings_file}.xml") else module_status ${modindex} 0 "branch-findbugs-${fn}-warnings.html" "${msg}" fi @@ -329,7 +342,7 @@ function findbugs_postinstall return 0 fi - big_console_header "Patch findbugs detection" + big_console_header "findbugs detection: ${BUILDMODE}" findbugs_runner patch @@ -463,7 +476,7 @@ function findbugs_rebuild { declare repostatus=$1 - if [[ "${repostatus}" = branch ]]; then + if [[ "${repostatus}" = branch || "${BUILDMODE}" = full ]]; then findbugs_preapply else findbugs_postinstall http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/gradle.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/gradle.sh b/precommit/test-patch.d/gradle.sh index 57e023d..e809465 100755 --- a/precommit/test-patch.d/gradle.sh +++ b/precommit/test-patch.d/gradle.sh @@ -106,9 +106,9 @@ function gradle_precompile if [[ "${repostatus}" = branch ]]; then # shellcheck disable=SC2153 - big_console_header "${PATCH_BRANCH} gradle bootstrap" + big_console_header "gradle boostrap: ${PATCH_BRANCH}" else - big_console_header "Patch gradle bootstrap" + big_console_header "gradle bootstrap: ${BUILDMODE}" fi personality_modules "${repostatus}" gradleboot @@ -277,5 +277,5 @@ function gradle_builtin_personality_file_tests function gradle_docker_support { - echo "-v ${HOME}/.gradle:${HOME}/.gradle" > "${PATCH_DIR}/buildtool-docker-params.txt" + DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${HOME}/.gradle:${HOME}/.gradle") } http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/htmlout.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/htmlout.sh b/precommit/test-patch.d/htmlout.sh new file mode 100755 index 0000000..bf6d06e --- /dev/null +++ b/precommit/test-patch.d/htmlout.sh @@ -0,0 +1,229 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +add_bugsystem htmlout + +## @description Usage info for htmlout plugin +## @audience private +## @stability evolving +## @replaceable no +function htmlout_usage +{ + yetus_add_option "--html-report-file=" "Save the final report to an HTML-formated file" +} + +## @description Option parsing for htmlout plugin +## @audience private +## @stability evolving +## @replaceable no +function htmlout_parse_args +{ + declare i + declare fn + + for i in "$@"; do + case ${i} in + --html-report-file=*) + fn=${i#*=} + ;; + esac + done + + if [[ -n "${fn}" ]]; then + touch "${fn}" 2>/dev/null + if [[ $? != 0 ]]; then + yetus_error "WARNING: cannot create ${fn}. Ignoring." + else + HTMLOUT_REPORTFILE=$(yetus_abs "${fn}") + fi + fi +} + +## @description Give access to the HTML report file in docker mode +## @audience private +## @stability evolving +## @replaceable no +function htmlout_docker_support +{ + if [[ -n ${HTMLOUT_REPORTFILE} ]]; then + DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${HTMLOUT_REPORTFILE}:${HTMLOUT_REPORTFILE}") + fi +} + + +## @description Write out an HTML version of the final report to a file +## @audience private +## @stability evolving +## @replaceable no +## @param runresult +function htmlout_finalreport +{ + declare result=$1 + declare i + declare commentfile="${HTMLOUT_REPORTFILE}" + declare comment + declare vote + declare ourstring + declare ela + declare subs + declare color + declare comment + + rm "${commentfile}" 2>/dev/null + + if [[ -z "${HTMLOUT_REPORTFILE}" ]]; then + return + fi + + big_console_header "Writing HTML to ${commentfile}" + + { + echo "" + + if [[ ${result} == 0 ]]; then + echo "" + else + echo "" + fi + echo "
+1 overall
-1 overall
" + echo "

" + } > "${commentfile}" + + i=0 + until [[ $i -eq ${#TP_HEADER[@]} ]]; do + ourstring=$(echo "${TP_HEADER[${i}]}" | tr -s ' ') + comment=$(echo "${ourstring}" | cut -f2 -d\|) + printf "%s\n" "${comment}" + ((i=i+1)) + done + + { + echo "" + echo "" + echo "" + echo "" + echo "" + echo "" + echo "" + } >> "${commentfile}" + + i=0 + until [[ $i -eq ${#TP_VOTE_TABLE[@]} ]]; do + ourstring=$(echo "${TP_VOTE_TABLE[${i}]}" | tr -s ' ') + vote=$(echo "${ourstring}" | cut -f2 -d\| | tr -d ' ') + subs=$(echo "${ourstring}" | cut -f3 -d\|) + ela=$(echo "${ourstring}" | cut -f4 -d\|) + comment=$(echo "${ourstring}" | cut -f5 -d\|) + + # summary line + if [[ -z ${vote} + && -n ${ela} ]]; then + color="black" + elif [[ -z ${vote} ]]; then + # keep same color + true + else + # new vote line + case ${vote} in + 1|"+1") + color="green" + ;; + -1) + color="red" + ;; + 0) + color="blue" + ;; + *) + color="black" + ;; + esac + fi + + { + echo "" + printf "" "${color}" "${vote}" + printf "" "${color}" "${subs}" + printf "" "${color}" "${ela}" + printf "" "${color}" "${comment}" + echo "" + } >> "${commentfile}" + ((i=i+1)) + done + { + echo "
VoteSubsystemRuntimeComment
%s%s%s%s
" + echo "

" + } >> "${commentfile}" + + if [[ ${#TP_TEST_TABLE[@]} -gt 0 ]]; then + { + echo "" + echo "" + echo "" + echo "" + echo "" + } >> "${commentfile}" + + i=0 + until [[ $i -eq ${#TP_TEST_TABLE[@]} ]]; do + ourstring=$(echo "${TP_TEST_TABLE[${i}]}" | tr -s ' ') + subs=$(echo "${ourstring}" | cut -f2 -d\|) + comment=$(echo "${ourstring}" | cut -f3 -d\|) + { + echo "" + printf "" "${color}" "${subs}" + printf "" "${color}" "${comment}" + echo "" + } >> "${commentfile}" + ((i=i+1)) + done + + { + echo "
ReasonTests
%s%s
" + echo "

" + } >> "${commentfile}" + fi + + { + echo "" + echo "" + echo "" + echo "" + echo "" + } >> "${commentfile}" + + i=0 + until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do + ourstring=$(echo "${TP_FOOTER_TABLE[${i}]}" | + ${SED} -e "s,@@BASE@@,${BUILD_URL}${BUILD_URL_ARTIFACTS},g" | + tr -s ' ') + subs=$(echo "${ourstring}" | cut -f2 -d\|) + comment=$(echo "${ourstring}" | cut -f3 -d\|) + { + echo "" + printf "" "${color}" "${subs}" + printf "" "${color}" "${comment}" + echo "" + } >> "${commentfile}" + ((i=i+1)) + done + { + echo "
SubsystemReport/Notes
%s%s
" + echo "

" + } >> "${commentfile}" + + printf "

This message was automatically generated.

" >> "${commentfile}" +} http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/maven.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/maven.sh b/precommit/test-patch.d/maven.sh index 2e006c1..3d37c37 100755 --- a/precommit/test-patch.d/maven.sh +++ b/precommit/test-patch.d/maven.sh @@ -95,6 +95,11 @@ function maven_parse_args function maven_initialize { + if ! verify_command "maven" "${MAVEN}"; then + return 1 + fi + + # we need to do this before docker does it as root maven_add_install mvneclipse maven_add_install mvnsite @@ -139,7 +144,7 @@ function maven_precheck fi if [[ ${MAVEN_CUSTOM_REPOS} = true ]]; then - MAVEN_LOCAL_REPO="${MAVEN_CUSTOM_REPOS_DIR}/${PROJECT_NAME}-${PATCH_BRANCH}-${INSTANCE}" + MAVEN_LOCAL_REPO="${MAVEN_CUSTOM_REPOS_DIR}/${PROJECT_NAME}-${PATCH_BRANCH}-${BUILDMODE}-${INSTANCE}" if [[ -e "${MAVEN_LOCAL_REPO}" && ! -d "${MAVEN_LOCAL_REPO}" ]]; then yetus_error "ERROR: ${MAVEN_LOCAL_REPO} is not a directory." @@ -369,7 +374,8 @@ function maven_builtin_personality_modules # this always makes sure the local repo has a fresh # copy of everything per pom rules. if [[ ${repostatus} == branch - && ${testtype} == mvninstall ]];then + && ${testtype} == mvninstall ]] || + [[ "${BUILDMODE}" = full ]];then personality_enqueue_module "${CHANGED_UNION_MODULES}" return fi @@ -450,9 +456,9 @@ function mvnsite_postcompile fi if [[ "${repostatus}" = branch ]]; then - big_console_header "Pre-patch ${PATCH_BRANCH} maven site verification" + big_console_header "maven site verification: ${PATCH_BRANCH}" else - big_console_header "Patch maven site verification" + big_console_header "maven site verification: ${BUILDMODE}" fi personality_modules "${repostatus}" mvnsite @@ -485,9 +491,9 @@ function mvneclipse_postcompile fi if [[ "${repostatus}" = branch ]]; then - big_console_header "Pre-patch ${PATCH_BRANCH} maven eclipse verification" + big_console_header "maven eclipse verification: ${PATCH_BRANCH}" else - big_console_header "Patch maven eclipse verification" + big_console_header "maven eclipse verification: ${BUILDMODE}" fi personality_modules "${repostatus}" mvneclipse @@ -537,9 +543,9 @@ function maven_precompile fi if [[ "${repostatus}" = branch ]]; then - big_console_header "Pre-patch ${PATCH_BRANCH} maven install" + big_console_header "maven install: ${PATCH_BRANCH}" else - big_console_header "Patch maven install" + big_console_header "maven install: ${BUILDMODE}" fi personality_modules "${repostatus}" mvninstall @@ -554,11 +560,10 @@ function maven_precompile function maven_docker_support { - echo "-v ${HOME}/.m2:${HOME}/.m2" > "${PATCH_DIR}/buildtool-docker-params.txt" + DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${HOME}/.m2:${HOME}/.m2") if [[ ${MAVEN_CUSTOM_REPOS} = true ]]; then - echo "-v ${MAVEN_CUSTOM_REPOS_DIR}:${MAVEN_CUSTOM_REPOS_DIR}" \ - >> "${PATCH_DIR}/buildtool-docker-params.txt" + DOCKER_EXTRAARGS=("${DOCKER_EXTRAARGS[@]}" "-v" "${MAVEN_CUSTOM_REPOS_DIR}:${MAVEN_CUSTOM_REPOS_DIR}") fi } @@ -676,5 +681,19 @@ function maven_reorder_modules yetus_debug "Maven: finish re-ordering modules" yetus_debug "Finished list: ${CHANGED_MODULES[*]}" - add_vote_table 0 mvndep "Maven dependency ordering for ${repostatus}" + # build some utility module lists for maven modules + for index in "${CHANGED_MODULES[@]}"; do + if [[ -d "${index}/src" ]]; then + MAVEN_SRC_MODULES=("${MAVEN_SRC_MODULES[@]}" "${index}") + if [[ -d "${index}/src/test" ]]; then + MAVEN_SRCTEST_MODULES=("${MAVEN_SRCTEST_MODULES[@]}" "${index}") + fi + fi + done + + if [[ "${BUILDMODE}" = patch ]]; then + add_vote_table 0 mvndep "Maven dependency ordering for ${repostatus}" + else + add_vote_table 0 mvndep "Maven dependency ordering" + fi } http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/perlcritic.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/perlcritic.sh b/precommit/test-patch.d/perlcritic.sh index 8297f46..291e94e 100755 --- a/precommit/test-patch.d/perlcritic.sh +++ b/precommit/test-patch.d/perlcritic.sh @@ -64,11 +64,11 @@ function perlcritic_preapply return 0 fi - big_console_header "Perl::Critic plugin: prepatch" + big_console_header "Perl::Critic plugin: ${PATCH_BRANCH}" start_clock - echo "Running perlcritic against modified perl scripts/modules." + echo "Running perlcritic against identified perl scripts/modules." pushd "${BASEDIR}" >/dev/null for i in "${CHANGED_FILES[@]}"; do if [[ ${i} =~ \.p[lm]$ && -f ${i} ]]; then @@ -106,7 +106,7 @@ function perlcritic_postapply return 0 fi - big_console_header "Perl::Critic plugin: postpatch" + big_console_header "Perl::Critic plugin: ${BUILDMODE}" start_clock @@ -114,7 +114,7 @@ function perlcritic_postapply # by setting the clock back offset_clock "${PERLCRITIC_TIMER}" - echo "Running perlcritic against modified perl scripts/modules." + echo "Running perlcritic against identified perl scripts/modules." # we re-check this in case one has been added pushd "${BASEDIR}" >/dev/null for i in "${CHANGED_FILES[@]}"; do @@ -147,11 +147,11 @@ function perlcritic_postapply statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" ) if [[ ${diffPostpatch} -gt 0 ]]; then - add_vote_table -1 perlcritic "The applied patch ${statstring}" + add_vote_table -1 perlcritic "${BUILDMODEMSG} ${statstring}" add_footer_table perlcritic "@@BASE@@/diff-patch-perlcritic.txt" return 1 elif [[ ${fixedpatch} -gt 0 ]]; then - add_vote_table +1 perlcritic "The applied patch ${statstring}" + add_vote_table +1 perlcritic "${BUILDMODEMSG} ${statstring}" return 0 fi http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/pylint.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/pylint.sh b/precommit/test-patch.d/pylint.sh index 2a9e1f8..7decb96 100755 --- a/precommit/test-patch.d/pylint.sh +++ b/precommit/test-patch.d/pylint.sh @@ -71,11 +71,11 @@ function pylint_preapply return 0 fi - big_console_header "pylint plugin: prepatch" + big_console_header "pylint plugin: ${PATCH_BRANCH}" start_clock - echo "Running pylint against modified python scripts." + echo "Running pylint against identified python scripts." pushd "${BASEDIR}" >/dev/null for i in "${CHANGED_FILES[@]}"; do if [[ ${i} =~ \.py$ && -f ${i} ]]; then @@ -87,7 +87,7 @@ function pylint_preapply if [[ -f ${PATCH_DIR}/${pylintStderr} ]]; then count=$(${GREP} -vc "^No config file found" "${PATCH_DIR}/${pylintStderr}") if [[ ${count} -gt 0 ]]; then - add_footer_table pylint "prepatch stderr: @@BASE@@/${pylintStderr}" + add_footer_table pylint "${PATCH_BRANCH} stderr: @@BASE@@/${pylintStderr}" return 1 fi fi @@ -113,7 +113,7 @@ function pylint_postapply return 0 fi - big_console_header "pylint plugin: postpatch" + big_console_header "pylint plugin: ${BUILDMODE}" start_clock @@ -121,7 +121,7 @@ function pylint_postapply # by setting the clock back offset_clock "${PYLINT_TIMER}" - echo "Running pylint against modified python scripts." + echo "Running pylint against identified python scripts." # we re-check this in case one has been added pushd "${BASEDIR}" >/dev/null for i in "${CHANGED_FILES[@]}"; do @@ -135,7 +135,7 @@ function pylint_postapply count=$(${GREP} -vc "^No config file found" "${PATCH_DIR}/${pylintStderr}") if [[ ${count} -gt 0 ]]; then add_vote_table -1 pylint "Something bad seems to have happened in running pylint. Please check pylint stderr files." - add_footer_table pylint "postpatch stderr: @@BASE@@/${pylintStderr}" + add_footer_table pylint "${BUILDMODEMSG} stderr: @@BASE@@/${pylintStderr}" return 1 fi fi @@ -158,11 +158,11 @@ function pylint_postapply statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" ) if [[ ${diffPostpatch} -gt 0 ]] ; then - add_vote_table -1 pylint "The applied patch ${statstring}" + add_vote_table -1 pylint "${BUILDMODEMSG} ${statstring}" add_footer_table pylint "@@BASE@@/diff-patch-pylint.txt" return 1 elif [[ ${fixedpatch} -gt 0 ]]; then - add_vote_table +1 pylint "The applied patch ${statstring}" + add_vote_table +1 pylint "${BUILDMODEMSG} ${statstring}" return 0 fi http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/rubocop.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/rubocop.sh b/precommit/test-patch.d/rubocop.sh index 8e1d5b8..828bea1 100755 --- a/precommit/test-patch.d/rubocop.sh +++ b/precommit/test-patch.d/rubocop.sh @@ -64,11 +64,11 @@ function rubocop_preapply return 0 fi - big_console_header "rubocop plugin: prepatch" + big_console_header "rubocop plugin: ${PATCH_BRANCH}" start_clock - echo "Running rubocop against modified ruby scripts." + echo "Running rubocop against identified ruby scripts." pushd "${BASEDIR}" >/dev/null for i in "${CHANGED_FILES[@]}"; do if [[ ${i} =~ \.rb$ && -f ${i} ]]; then @@ -106,7 +106,7 @@ function rubocop_postapply return 0 fi - big_console_header "rubocop plugin: postpatch" + big_console_header "rubocop plugin: ${BUILDMODE}" start_clock @@ -114,7 +114,7 @@ function rubocop_postapply # by setting the clock back offset_clock "${RUBOCOP_TIMER}" - echo "Running rubocop against modified ruby scripts." + echo "Running rubocop against identified ruby scripts." # we re-check this in case one has been added pushd "${BASEDIR}" >/dev/null for i in "${CHANGED_FILES[@]}"; do @@ -146,11 +146,11 @@ function rubocop_postapply statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" ) if [[ ${diffPostpatch} -gt 0 ]] ; then - add_vote_table -1 rubocop "The applied patch ${statstring}" + add_vote_table -1 rubocop "${BUILDMODEMSG} ${statstring}" add_footer_table rubocop "@@BASE@@/diff-patch-rubocop.txt" return 1 elif [[ ${fixedpatch} -gt 0 ]]; then - add_vote_table +1 rubocop "The applied patch ${statstring}" + add_vote_table +1 rubocop "${BUILDMODEMSG} ${statstring}" return 0 fi http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/ruby-lint.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/ruby-lint.sh b/precommit/test-patch.d/ruby-lint.sh index 94d4f53..78be7b6 100755 --- a/precommit/test-patch.d/ruby-lint.sh +++ b/precommit/test-patch.d/ruby-lint.sh @@ -63,11 +63,11 @@ function ruby_lint_preapply return 0 fi - big_console_header "ruby-lint plugin: prepatch" + big_console_header "ruby-lint plugin: ${PATCH_BRANCH}" start_clock - echo "Running ruby-lint against modified ruby scripts." + echo "Running ruby-lint against identified ruby scripts." pushd "${BASEDIR}" >/dev/null for i in "${CHANGED_FILES[@]}"; do if [[ ${i} =~ \.rb$ && -f ${i} ]]; then @@ -136,7 +136,7 @@ function ruby_lint_postapply return 0 fi - big_console_header "ruby-lint plugin: postpatch" + big_console_header "ruby-lint plugin: ${BUILDMODE}" start_clock @@ -144,7 +144,7 @@ function ruby_lint_postapply # by setting the clock back offset_clock "${RUBY_LINT_TIMER}" - echo "Running ruby-lint against modified ruby scripts." + echo "Running ruby-lint against identified ruby scripts." # we re-check this in case one has been added pushd "${BASEDIR}" >/dev/null for i in "${CHANGED_FILES[@]}"; do @@ -176,11 +176,11 @@ function ruby_lint_postapply statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" ) if [[ ${diffPostpatch} -gt 0 ]] ; then - add_vote_table -1 ruby-lint "The applied patch ${statstring}" + add_vote_table -1 ruby-lint "${BUILDMODEMSG} ${statstring}" add_footer_table ruby-lint "@@BASE@@/diff-patch-ruby-lint.txt" return 1 elif [[ ${fixedpatch} -gt 0 ]]; then - add_vote_table +1 ruby-lint "The applied patch ${statstring}" + add_vote_table +1 ruby-lint "${BUILDMODEMSG} ${statstring}" return 0 fi http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/shellcheck.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/shellcheck.sh b/precommit/test-patch.d/shellcheck.sh index e9a7e15..0e053e3 100755 --- a/precommit/test-patch.d/shellcheck.sh +++ b/precommit/test-patch.d/shellcheck.sh @@ -106,7 +106,7 @@ function shellcheck_preapply return 0 fi - big_console_header "shellcheck plugin: prepatch" + big_console_header "shellcheck plugin: ${PATCH_BRANCH}" start_clock @@ -157,7 +157,7 @@ function shellcheck_postapply return 0 fi - big_console_header "shellcheck plugin: postpatch" + big_console_header "shellcheck plugin: ${BUILDMODE}" start_clock @@ -194,12 +194,12 @@ function shellcheck_postapply statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" ) if [[ ${diffPostpatch} -gt 0 ]] ; then - add_vote_table -1 shellcheck "The applied patch ${statstring}" + add_vote_table -1 shellcheck "${BUILDMODEMSG} ${statstring}" add_footer_table shellcheck "@@BASE@@/diff-patch-shellcheck.txt" bugsystem_linecomments "shellcheck" "${PATCH_DIR}/diff-patch-shellcheck.txt" return 1 elif [[ ${fixedpatch} -gt 0 ]]; then - add_vote_table +1 shellcheck "The applied patch ${statstring}" + add_vote_table +1 shellcheck "${BUILDMODEMSG} ${statstring}" return 0 fi http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/shelldocs.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/shelldocs.sh b/precommit/test-patch.d/shelldocs.sh index fea982a..d52fa6b 100755 --- a/precommit/test-patch.d/shelldocs.sh +++ b/precommit/test-patch.d/shelldocs.sh @@ -103,7 +103,7 @@ function shelldocs_preapply return 0 fi - big_console_header "shelldocs plugin: prepatch" + big_console_header "shelldocs plugin: ${PATCH_BRANCH}" start_clock @@ -134,7 +134,7 @@ function shelldocs_postapply return 0 fi - big_console_header "shelldocs plugin: postpatch" + big_console_header "shelldocs plugin: ${BUILDMODE}" start_clock @@ -170,12 +170,12 @@ function shelldocs_postapply statstring=$(generic_calcdiff_status "${numPrepatch}" "${numPostpatch}" "${diffPostpatch}" ) if [[ ${diffPostpatch} -gt 0 ]] ; then - add_vote_table -1 shelldocs "The applied patch ${statstring}" + add_vote_table -1 shelldocs "${BUILDMODEMSG} ${statstring}" add_footer_table shelldocs "@@BASE@@/diff-patch-shelldocs.txt" bugsystem_linecomments "shelldocs" "${PATCH_DIR}/diff-patch-shelldocs.txt" return 1 elif [[ ${fixedpatch} -gt 0 ]]; then - add_vote_table +1 shelldocs "The applied patch ${statstring}" + add_vote_table +1 shelldocs "${BUILDMODEMSG} ${statstring}" return 0 fi http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/test4tests.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/test4tests.sh b/precommit/test-patch.d/test4tests.sh index 644fbf9..6c3230c 100755 --- a/precommit/test-patch.d/test4tests.sh +++ b/precommit/test-patch.d/test4tests.sh @@ -27,6 +27,10 @@ function test4tests_patchfile declare testReferences=0 declare i + if [[ "${BUILDMODE}" = full ]]; then + return + fi + big_console_header "Checking there are new or changed tests in the patch." if ! verify_needed_test unit; then http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/whitespace.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/whitespace.sh b/precommit/test-patch.d/whitespace.sh index a613d81..da0789b 100755 --- a/precommit/test-patch.d/whitespace.sh +++ b/precommit/test-patch.d/whitespace.sh @@ -49,21 +49,37 @@ function whitespace_postcompile start_clock pushd "${BASEDIR}" >/dev/null - # shellcheck disable=SC2016 - ${AWK} '/\t/ {print $0}' \ - "${GITDIFFCONTENT}" \ - | ${GREP} -v Makefile: >> "${PATCH_DIR}/whitespace-tabs.txt" - ${GREP} -E '[[:blank:]]$' \ - "${GITDIFFCONTENT}" \ - >> "${PATCH_DIR}/whitespace-eol.txt" + case "${BUILDMODE}" in + patch) + # shellcheck disable=SC2016 + ${AWK} '/\t/ {print $0}' \ + "${GITDIFFCONTENT}" \ + | ${GREP} -v Makefile: >> "${PATCH_DIR}/whitespace-tabs.txt" + + ${GREP} -E '[[:blank:]]$' \ + "${GITDIFFCONTENT}" \ + >> "${PATCH_DIR}/whitespace-eol.txt" + ;; + full) + ${GIT} grep -n -I --extended-regexp '[[:blank:]]$' \ + >> "${PATCH_DIR}/whitespace-eol.txt" + ${GIT} grep -n -I $'\t' \ + | "${GREP}" -v Makefile \ + >> "${PATCH_DIR}/whitespace-tabs.txt" + ;; + esac # shellcheck disable=SC2016 count=$(wc -l "${PATCH_DIR}/whitespace-eol.txt" | ${AWK} '{print $1}') if [[ ${count} -gt 0 ]]; then - add_vote_table -1 whitespace "The patch has ${count}"\ - " line(s) that end in whitespace. Use git apply --whitespace=fix." + if [[ "${BUILDMODE}" = full ]]; then + add_vote_table -1 whitespace "${BUILDMODEMSG} has ${count} line(s) that end in whitespace." + else + add_vote_table -1 whitespace \ + "${BUILDMODEMSG} has ${count} line(s) that end in whitespace. Use git apply --whitespace=fix." + fi whitespace_linecomment_reporter "${PATCH_DIR}/whitespace-eol.txt" "end of line" add_footer_table whitespace "@@BASE@@/whitespace-eol.txt" @@ -74,7 +90,7 @@ function whitespace_postcompile count=$(wc -l "${PATCH_DIR}/whitespace-tabs.txt" | ${AWK} '{print $1}') if [[ ${count} -gt 0 ]]; then - add_vote_table -1 whitespace "The patch has ${count}"\ + add_vote_table -1 whitespace "${BUILDMODEMSG} ${count}"\ " line(s) with tabs." add_footer_table whitespace "@@BASE@@/whitespace-tabs.txt" whitespace_linecomment_reporter "${PATCH_DIR}/whitespace-tabs.txt" "tabs in line" @@ -87,6 +103,6 @@ function whitespace_postcompile fi popd >/dev/null - add_vote_table +1 whitespace "Patch has no whitespace issues." + add_vote_table +1 whitespace "${BUILDMODEMSG} has no whitespace issues." return 0 } http://git-wip-us.apache.org/repos/asf/yetus/blob/612a0f2d/precommit/test-patch.d/xml.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/xml.sh b/precommit/test-patch.d/xml.sh index 08e0a61..6460522 100755 --- a/precommit/test-patch.d/xml.sh +++ b/precommit/test-patch.d/xml.sh @@ -48,7 +48,7 @@ function xml_postcompile return 0 fi - big_console_header "Checking if XML files are well-formed" + big_console_header "XML verification: ${BUILDMODE}" js="${JAVA_HOME}/bin/jrunscript" @@ -65,13 +65,13 @@ function xml_postcompile done if [[ ${count} -gt 0 ]]; then - add_vote_table -1 xml "The patch has ${count} ill-formed XML file(s)." + add_vote_table -1 xml "${BUILDMODEMSG} has ${count} ill-formed XML file(s)." add_footer_table xml "@@BASE@@/xml.txt" popd >/dev/null return 1 fi popd >/dev/null - add_vote_table +1 xml "The patch has no ill-formed XML file." + add_vote_table +1 xml "${BUILDMODEMSG} has no ill-formed XML file." return 0 }