From commits-return-1317-archive-asf-public=cust-asf.ponee.io@yetus.apache.org Mon Apr 22 16:36:10 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id B5DC018064D for ; Mon, 22 Apr 2019 18:36:08 +0200 (CEST) Received: (qmail 80227 invoked by uid 500); 22 Apr 2019 16:36:08 -0000 Mailing-List: contact commits-help@yetus.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@yetus.apache.org Delivered-To: mailing list commits@yetus.apache.org Received: (qmail 80217 invoked by uid 99); 22 Apr 2019 16:36:08 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Apr 2019 16:36:08 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id 49DD887A36; Mon, 22 Apr 2019 16:36:07 +0000 (UTC) Date: Mon, 22 Apr 2019 16:36:07 +0000 To: "commits@yetus.apache.org" Subject: [yetus] branch master updated: YETUS-856. Add markdownlint support (#47) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <155595096715.31400.107659905540524472@gitbox.apache.org> From: aw@apache.org X-Git-Host: gitbox.apache.org X-Git-Repo: yetus X-Git-Refname: refs/heads/master X-Git-Reftype: branch X-Git-Oldrev: 5cf772d48f7d062e18886199eb0390772a412fa7 X-Git-Newrev: 37c8a5f24e006de26d8cb86b1a2d4ccb7939ca5d X-Git-Rev: 37c8a5f24e006de26d8cb86b1a2d4ccb7939ca5d X-Git-NotificationType: ref_changed_plus_diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated This is an automated email from the ASF dual-hosted git repository. aw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/yetus.git The following commit(s) were added to refs/heads/master by this push: new 37c8a5f YETUS-856. Add markdownlint support (#47) 37c8a5f is described below commit 37c8a5f24e006de26d8cb86b1a2d4ccb7939ca5d Author: Allen Wittenauer AuthorDate: Mon Apr 22 09:36:02 2019 -0700 YETUS-856. Add markdownlint support (#47) --- .../coprocs.d/README.md => .markdownlint.yaml | 14 +- README.md | 14 +- asf-site-src/Gemfile.lock | 32 +- asf-site-src/source/contribute.html.md | 18 +- asf-site-src/source/contribute/releases.md | 528 +++++++++++---------- asf-site-src/source/contribute/website.md | 20 +- .../in-progress/interface-classification.md | 224 ++++----- .../documentation/in-progress/precommit-admin.md | 34 +- .../in-progress/precommit-advanced.md | 52 +- .../documentation/in-progress/precommit-basic.md | 17 +- .../in-progress/precommit-bugsystems.md | 47 +- .../in-progress/precommit-buildtools.md | 114 ++--- .../in-progress/precommit-docker-cleanup.md | 9 +- .../documentation/in-progress/precommit-docker.md | 3 +- .../in-progress/precommit-patchnames.md | 34 +- .../documentation/in-progress/precommit-qbt.md | 27 +- .../documentation/in-progress/precommit-robots.md | 106 ++--- .../in-progress/precommit-testformats.md | 21 +- .../documentation/in-progress/releasedocmaker.md | 50 +- .../source/documentation/in-progress/shelldocs.md | 7 +- .../in-progress/yetus-maven-plugin.md | 15 +- asf-site-src/source/index.html.md | 1 + asf-site-src/source/mailinglists.html.md | 2 + asf-site-src/source/yetus-docker-image.md | 28 +- .../src/main/shell/coprocs.d/{README.md => README} | 3 +- .../src/main/shell/test-patch-docker/Dockerfile | 4 +- .../src/main/shell/test-patch.d/markdownlint.sh | 166 +++++++ 27 files changed, 913 insertions(+), 677 deletions(-) diff --git a/precommit/src/main/shell/coprocs.d/README.md b/.markdownlint.yaml old mode 100755 new mode 100644 similarity index 75% copy from precommit/src/main/shell/coprocs.d/README.md copy to .markdownlint.yaml index e960cad..d886982 --- a/precommit/src/main/shell/coprocs.d/README.md +++ b/.markdownlint.yaml @@ -1,4 +1,3 @@ -#!/usr/bin/env bash # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -14,12 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -############################# -## -## bash v4+ coproc routines -## -############################# - -# bash v3 and lower will treat coproc commands as syntax errors -# therefore, ALL functions which activate coprocs are located -# here \ No newline at end of file +--- +default: true +line-length: false +commands-show-output: false diff --git a/README.md b/README.md index 108fd19..88f3974 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ specific language governing permissions and limitations under the License. --> + # Apache Yetus Apache Yetus is a collection of libraries and tools that enable @@ -27,14 +28,19 @@ Here is a list of the major components: * [Website source](asf-site-src/) Holds our documentation, which is presented via [our website](https://yetus.apache.org/). + * [Precommit](precommit/) Precommit provides robust tools to deal with contributions, including applying patches from a variety of project sources and evaluating them against project norms via a system of plugins. See the [precommit overview](asf-site-src/source/documentation/in-progress/precommit-architecture.md) to get started working with precommit. + * [Audience Annotations](audience-annotations-component/) Audience Annotations allows projects to use Java Annotations to delineate public and non-public parts of their APIs. It also provides doclets to generate javadocs filtered by the intended audience. Currently builds with Maven 3.2.0+. + * [Shelldocs](shelldocs/) Shelldocs processes comments on Bash functions for annotations similar to Javadoc. It also includes built-in audience scoping functionality similar to the doclet from Audience Annotations. + * [Release Doc Maker](release-doc-maker/) Release Doc Maker analyzes Jira and Git information to produce Markdown formatted release notes. + * [yetus-maven-plugin](yetus-maven-plugin/) Builds a maven plugin that provides some small utilities for some uncommon maven requirements (such as symlinks) in addition to being mavenized versions of some of the Apache Yetus functionality. @@ -43,7 +49,8 @@ Builds a maven plugin that provides some small utilities for some uncommon maven For full instructions on how to build releases and the website, see the [guide to contributing](asf-site-src/source/contribute.html.md) for requirements and instructions. ```bash -# Launch a Docker container that has all of the project's dependencies and a working build environment +# Launch a Docker container that has all of the project's dependencies +# and a working build environment ./start-build-env.sh # Build the binary tarball, located in yetus-dist/target/artifacts: @@ -52,7 +59,8 @@ mvn clean install # Build the binary and source tarballs and sign the content: mvn clean install -Papache-release -# Same, but if outside the container and need to let the system know that the OS uses 'gpg2' instead of 'gpg': +# Same, but if outside the container and need to let the system know +# that the OS uses 'gpg2' instead of 'gpg': mvn clean install -Papache-release -Pgpg2 # Build the binary and source tarballs, but skip signing them: @@ -66,7 +74,7 @@ After executing one or more of the Apache Maven commands, artifacts will be in ` ## Container Quickstart -The project makes available two convenience container images on https://hub.docker.com for both tagged releases and for the master branch. It is highly recommended that casual users use a tagged release so as to not be surprised by incompatible changes that are still rolling through the master branch. +The project makes available two convenience container images on for both tagged releases and for the master branch. It is highly recommended that casual users use a tagged release so as to not be surprised by incompatible changes that are still rolling through the master branch. ### apache/yetus-base diff --git a/asf-site-src/Gemfile.lock b/asf-site-src/Gemfile.lock index 293c203..072bb37 100644 --- a/asf-site-src/Gemfile.lock +++ b/asf-site-src/Gemfile.lock @@ -1,5 +1,5 @@ GIT - remote: https://github.com/middleman/middleman-syntax.git + remote: git://github.com/middleman/middleman-syntax.git revision: 0377cc9d2219c2a6ffc2acfd39057f833934af9a specs: middleman-syntax (3.1.0) @@ -7,15 +7,15 @@ GIT rouge (~> 3.1) GIT - remote: https://github.com/vmg/redcarpet.git - revision: 92a7b3ae2241b862e9bf45e0af3cf53ebdfb0afb + remote: git://github.com/vmg/redcarpet.git + revision: 94f6e27bdf2395efa555a7c772a3d8b70fb84346 specs: redcarpet (3.4.0) GEM remote: https://rubygems.org/ specs: - activesupport (4.2.10) + activesupport (4.2.11.1) i18n (~> 0.7) minitest (~> 5.1) thread_safe (~> 0.3, >= 0.3.4) @@ -26,7 +26,7 @@ GEM rack (>= 1.0.0) rack-test (>= 0.5.4) xpath (~> 2.0) - chunky_png (1.3.10) + chunky_png (1.3.11) coffee-script (2.4.1) coffee-script-source execjs @@ -49,7 +49,7 @@ GEM erubis (2.7.0) eventmachine (1.2.7) execjs (2.7.0) - ffi (1.9.25) + ffi (1.10.0) haml (5.0.4) temple (>= 0.8.0) tilt @@ -58,7 +58,7 @@ GEM uber (~> 0.0.14) http_parser.rb (0.6.0) i18n (0.7.0) - json (2.1.0) + json (2.2.0) kramdown (1.17.0) libv8 (3.16.14.19) listen (3.0.8) @@ -98,27 +98,27 @@ GEM sprockets-sass (~> 1.3.0) mime-types (3.2.2) mime-types-data (~> 3.2015) - mime-types-data (3.2018.0812) - mini_portile2 (2.3.0) + mime-types-data (3.2019.0331) + mini_portile2 (2.4.0) minitest (5.11.3) multi_json (1.13.1) - nokogiri (1.8.5) - mini_portile2 (~> 2.3.0) + nokogiri (1.10.2) + mini_portile2 (~> 2.4.0) padrino-helpers (0.12.9) i18n (~> 0.6, >= 0.6.7) padrino-support (= 0.12.9) tilt (>= 1.4.1, < 3) padrino-support (0.12.9) activesupport (>= 3.1) - rack (1.6.10) + rack (1.6.11) rack-livereload (0.3.17) rack rack-test (1.1.0) rack (>= 1.0, < 3) rake (10.3.1) rb-fsevent (0.10.3) - rb-inotify (0.9.10) - ffi (>= 0.5.0, < 2) + rb-inotify (0.10.0) + ffi (~> 1.0) ref (2.0.0) rouge (3.3.0) sass (3.4.25) @@ -132,11 +132,11 @@ GEM sprockets-sass (1.3.1) sprockets (~> 2.0) tilt (~> 1.1) - temple (0.8.0) + temple (0.8.1) therubyracer (0.12.3) libv8 (~> 3.16.14.15) ref - thor (0.20.0) + thor (0.20.3) thread_safe (0.3.6) tilt (1.4.1) tzinfo (1.2.5) diff --git a/asf-site-src/source/contribute.html.md b/asf-site-src/source/contribute.html.md index 8f1633f..b8d9801 100644 --- a/asf-site-src/source/contribute.html.md +++ b/asf-site-src/source/contribute.html.md @@ -17,23 +17,23 @@ under the License. --> -#How To Contribute +# How To Contribute ## Submitting Changes We use git as our version control system. To streamline the process of giving proper credit to the contributors when committing patches, we encourage contributors to submit patches generated using git format-patch. This has many benefits: - * Committers can't forget to attribute proper credit to the contributor - * The contributors name and email address shows up in git log - * When viewing Yetus's source code on https://github.com/apache/yetus , the commits from the contributor are linked to their github.com account if it's linked to the same email address they used when generating the git format-patch +* Committers can't forget to attribute proper credit to the contributor +* The contributors name and email address shows up in git log +* When viewing Yetus's source code on , the commits from the contributor are linked to their github.com account if it's linked to the same email address they used when generating the git format-patch Long story short, it makes both the contributors' and committers' lives easier, so please generate your patches using git format-patch. Here are some instructions on how to generate patches: - * Ensure that you have all of your change as 1 commit which has the correct commit message - something like `YETUS-1. Update shellcheck plug-in to support bats files` - * Then run a command like: `git format-patch HEAD^..HEAD --stdout > YETUS-1.00.patch` - * Upload the YETUS-1.00.patch file to the aforementioned JIRA +* Ensure that you have all of your change as 1 commit which has the correct commit message - something like `YETUS-1. Update shellcheck plug-in to support bats files` +* Then run a command like: `git format-patch HEAD^..HEAD --stdout > YETUS-1.00.patch` +* Upload the YETUS-1.00.patch file to the aforementioned JIRA The naming of the patch should be in (JIRA).(patch number).patch or, if it needs to apply to a specific branch, (JIRA).(branch name).(patch number).patch format. For example, YETUS-9.00.patch, YETUS-500.02.patch, or YETUS-23.cmake.11.patch. This way, if you need to upload another version of the patch, you should keep the file name the same and JIRA will sort them according to date/time if multiple files have the same name. This feature is also useful to traceback the history of a patch a [...] @@ -41,5 +41,5 @@ The naming of the patch should be in (JIRA).(patch number).patch or, if it needs Below are guides meant to give you help accomplishing specific tasks for the project: - * [Maintaining the Yetus Website](website) - walks through how to view the website locally, update various static and generated pages, and render the html for publishing. - * [Working with Release Candiates](releases) - covers managing the release process, validating proposed release candidates, and publishing project approved artifacts. +* [Maintaining the Yetus Website](website) - walks through how to view the website locally, update various static and generated pages, and render the html for publishing. +* [Working with Release Candiates](releases) - covers managing the release process, validating proposed release candidates, and publishing project approved artifacts. diff --git a/asf-site-src/source/contribute/releases.md b/asf-site-src/source/contribute/releases.md index 4654848..f95ef57 100644 --- a/asf-site-src/source/contribute/releases.md +++ b/asf-site-src/source/contribute/releases.md @@ -1,4 +1,4 @@ - - +# Managing a Release -* [Managing a Release](#managing-a-release) - * [Dependencies](#dependencies) - * [Setup](#setup) - * [Release Candidate\(s\)](#release-candidates) - * [Verification](#verification) - * [Cleanup](#cleanup) + - +- [Dependencies](#dependencies) +- [Setup](#setup) +- [Release Candidate\(s\)](#release-candidates) +- [Verification](#verification) +- [Cleanup](#cleanup) -# Managing a Release + The Apache Yetus community encourages all committers to help on driving releases. To that end, this section seeks to outline the tools and process you'll use when managing a release. Note that these are our community norms; they do not supersede foundation policy should the two disagree. @@ -78,11 +77,11 @@ Before attempting to do a release, verify that the documentation, website, etc, ### Ensure Your Public Key is in KEYS -Like many ASF projects, we provide a single file that downstream folks can use to verify our release artifacts. It's located in the project's distribution area: https://www.apache.org/dist/yetus/KEYS. You can read about this file in the ASF guide to release signing's section [The KEYS File](https://www.apache.org/dist/yetus/KEYS). If your public key is not already included in this file, you will need to add it. You can either follow the instructions in the previously mentioned guide or t [...] +Like many ASF projects, we provide a single file that downstream folks can use to verify our release artifacts. It's located in the project's distribution area . You can read about this file in the [ASF guide to release signing](https://www.apache.org/dev/release-signing) section. If your public key is not already included in [the KEYS file](https://www.apache.org/dist/yetus/KEYS), you will need to add it. You can either follow the instructions in the p [...] Example commands: -``` +```bash $ svn co https://dist.apache.org/repos/dist/release/yetus yetus-dist-release $ cd yetus-dist-release $ (gpg --list-sigs && gpg --armor --export ) >> KEYS @@ -94,32 +93,27 @@ $ svn commit -m "Added myself to KEYS." Like the rest of our project activity, we'll use an issue in JIRA to track managing the release. You should create this issue and assign it to yourself. As you make your way through the process of creating and running votes on release candidates, this issue will give you a centralized place to collect pointers to your work. -1. Browse to the ASF JIRA instance's "create issue" page: https://issues.apache.org/jira/secure/CreateIssue!default.jspa -1. Select "Yetus" for the Project and "Task" for the issue type. Click "Next" +1. Browse to the ASF JIRA instance's "create issue" page: +1. Select "Yetus" for the Project and "Task" for the issue type. Click "Next". 1. On the next screen, use a subject line like "Release VERSION", with an appropriate version number. Fill in the following fields and click "Create". - - The component should be "website and documentation" - - Affects Version and Fix Version should both be the version you are releasing - - Assignee should be you - - Add a description similar to "Generate release candidates as needed to create a VERSION release." with an appropriate version number. - -Next, create a shortened link to the JIRA version's release notes. This link should use the ASF link shortener, https://s.apache.org/. To find the appropriate release notes page: - -1. Browse to the Yetus JIRA versions page: https://issues.apache.org/jira/browse/YETUS/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel -1. Click on the Name of the release you are managing -1. Click on the "Release Notes" button. If it isn't shown, click on the "Summary" button in the left menu -1. Copy this URL -1. Browse to the ASF URL shortener: https://s.apache.org/ + - The component should be "website and documentation" + - Affects Version and Fix Version should both be the version you are releasing + - Assignee should be you + - Add a description similar to "Generate release candidates as needed to create a VERSION release." with an appropriate version number. +1. Next, create a shortened link to the JIRA version's release notes. This link should use the ASF link shortener, . To find the appropriate release notes page: + - Browse to the Yetus JIRA versions page: + - Click on the Name of the release you are managing + - Click on the "Release Notes" button. If it isn't shown, click on the "Summary" button in the left menu + - Copy this URL +1. Browse to the ASF URL shortener: 1. Paste the URL into the "URI" field 1. Set the optional key field to 'yetus-_version_-jira' - -For example, on the 0.7.0 release, you would use 'https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318920&version=12334330' for the URI field and 'yetus-0.7.0-jira' for the key. - -Finally, you should create a JIRA version that matches the release _following_ the one you are managing. This action is so that folks can continue to work on things that won't make it into the in-progress release while we evaluate candidates. - -1. Browse to the ASF JIRA project management page for versions: https://issues.apache.org/jira/plugins/servlet/project-config/YETUS/versions -1. Fill in a version one minor version up from the release you're managing. E.g., when managing the 0.7.0 release, fill in 0.3.0. -1. Set a start date of today. -1. Click "Add" + For example, on the 0.7.0 release, you would use `https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318920&version=12334330` for the URI field and 'yetus-0.7.0-jira' for the key. +1. Finally, you should create a JIRA version that matches the release _following_ the one you are managing. This action is so that folks can continue to work on things that won't make it into the in-progress release while we evaluate candidates. + 1. Browse to the ASF JIRA project management page for versions: + 1. Fill in a version one minor version up from the release you're managing. E.g., when managing the 0.7.0 release, fill in 0.3.0. + 1. Set a start date of today. + 1. Click "Add" ### Work in Git @@ -127,7 +121,7 @@ Once you have an issue to track things, you can create the git branch for stagin Example commands, presuming the release under management is **0.7.0** and the JIRA issue is **YETUS-XXX**: -``` +```bash $ # Ensure master is up to date $ mvn clean $ git fetch origin @@ -163,14 +157,13 @@ yetus-minimaven-plugin/pom.xml At this point, you should edit the files mentioned above. They must have the version we expect upon a successful release. Search for instances of *VERSION-SNAPSHOT* and replace with *VERSION*; e.g., *0.7.0-SNAPSHOT* should become *0.7.0*: -``` +```bash $ perl -pi -e 's,0.7.0-SNAPSHOT,0.7.0,g' $(find . -type f) ``` +After you are done, create a branch-specific patch and then prepare to update the master branch. - After you are done, create a branch-specific patch and then prepare to update the master branch. - -``` +```bash $ git add -p $ git commit -m "YETUS-XXX. Stage version 0.7.0." $ git format-patch --stdout origin/YETUS-XXX > path/to/patches/YETUS-XXX-YETUS-XXX.1.patch @@ -191,14 +184,13 @@ yetus-minimaven-plugin/pom.xml Now update these files, but this time you should update them for the next minor version's SNAPSHOT. e.g., *0.7.0-SNAPSHOT* should become *0.8.0-SNAPSHOT*: - -``` +```bash $ perl -pi -e 's,0.7.0-SNAPSHOT,0.8.0-SNAPSHOT,g' $(find . -type f) ``` After you are done, create a patch. -``` +```bash $ git add -p $ git commit -m "YETUS-XXX. bump master version to 0.8.0-SNAPSHOT" $ git format-patch --stdout origin/master > path/to/patches/YETUS-XXX.1.patch @@ -210,35 +202,48 @@ Both of these patch files should be uploaded to your release issue for review. P Depending on how candidate evaluation goes, you may end up performing these steps multiple times. Before you start, you'll need to decide when you want each candidate's vote thread to end. ASF policy requires a minimum voting period of 72 hours (ref [ASF Voting Policy](https://www.apache.org/foundation/voting.html)), so you should ensure enough padding to complete the candidate generation process in time. Ideally, you would plan to post the vote thread on a Friday morning (US time) with [...] -1. Update JIRA version release date. Browse to the JIRA project version management page (https://issues.apache.org/jira/plugins/servlet/project-config/YETUS/versions), mark the version as 'Release', and set the release date. Our generated release notes will use this date. -1. Update your `${HOME}/.m2/settings.xml` file to include the Maven snapshot information as indicated on https://www.apache.org/dev/publishing-maven-artifacts.html +1. Update JIRA version release date. Browse to the JIRA project version management page , mark the version as 'Release', and set the release date. Our generated release notes will use this date. +1. Update your `${HOME}/.m2/settings.xml` file to include the Maven snapshot information as indicated on 1. Build release artifacts. You should use our convenience script to create the tarballs and markdown documents for a release. Run the following from the release staging branch and inspect the results: - $ mvn --batch-mode clean deploy -Papache-release - $ mvn --batch-mode site site:stage - $ ls -lah yetus-dist/target/artifacts/* -1. Check out the staging area for release candidates and make a directory for this candidate, somewhere outside of your working directory. Copy the artifacts (**except for the site.tar.gz**) from the previous step into place. For example, when working on RC1 for the 0.7.0 release + ```bash + $ mvn --batch-mode clean deploy -Papache-release + $ mvn --batch-mode site site:stage + $ ls -lah yetus-dist/target/artifacts/* + ``` + +1. Check out the staging area for release candidates and make a directory for this candidate, somewhere outside of your working directory. Copy the artifacts from the previous step into place. For example, when working on RC1 for the 0.7.0 release + + ```bash + $ svn co `https://dist.apache.org/repos/dist/dev/yetus/` yetus-dist-dev + $ cd yetus-dist-dev + $ mkdir 0.7.0-RC1 + $ cd 0.7.0-RC1 + $ cp path/to/yetus/target/RELEASENOTES.md path/to/yetus/target/CHANGELOG.md path/to/yetus/target/*.tar.gz . + ``` - $ svn co https://dist.apache.org/repos/dist/dev/yetus/ yetus-dist-dev - $ cd yetus-dist-dev - $ mkdir 0.7.0-RC1 - $ cd 0.7.0-RC1 - $ cp path/to/yetus/target/RELEASENOTES.md path/to/yetus/target/CHANGELOG.md path/to/yetus/target/*.tar.gz . 1. While still in the staging area, sign the artifacts and create the needed checksum files: - $ for artifact in *; do - echo ${artifact} - gpg --use-agent --armor --output "${artifact}".asc --detach-sig "${artifact}" - gpg --print-mds "${artifact}" >"${artifact}".mds - shasum -a 512 "${artifact}" >"${artifact}".sha512 - done + ```bash + $ for artifact in *; do + echo ${artifact} + gpg --use-agent --armor --output "${artifact}".asc --detach-sig "${artifact}" + gpg --print-mds "${artifact}" >"${artifact}".mds + shasum -a 512 "${artifact}" >"${artifact}".sha512 + done + ``` + 1. Push the release candidate to staging distribution. This will make the artifacts visible for the vote. - $ cd .. - $ svn add 0.7.0-RC1 - $ svn commit -m "stage Apache Yetus 0.7.0-RC1" -Afterward, the artifacts should be visible via the web under the same URL used when checking out. In the case of 0.7.0-RC1: https://dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/ + ```bash + $ cd .. + $ svn add 0.7.0-RC1 + $ svn commit -m "stage Apache Yetus 0.7.0-RC1" + Afterward, the artifacts should be visible via the web under the same URL used when checking out. In the case of 0.7.0-RC1: + ``` + 1. Examine staged maven build. Go to the [ASF repository](https://repository.apache.org/) and log in with your asf LDAP credentials. Look for the staging repository with a name that includes "yetus". Clicking on it will give you a link to an "Open" repository. You can examine the structure in the Nexus API while you're logged in. If it looks essentially correct, "Close" the repository. Refreshing and clicking on the repository will give you a link in the Summary tab that other folks can [...] +1. Create a short link that should point to some online timezone conversion utility that will point to when the vote will end. ASF votes often use timeanddate.com's Event Time Announcer: . 1. Call a vote on the release candidate. At this point, you have everything you need to call a vote. Your vote thread must contain "[VOTE]" in the subject line, a link to the candidate staging area you created, a source repository commit hash, and voting rules. It should also contain hashes for the artifacts. Here is an example draft for 0.7.0-RC1, update it as appropriate for your release: Subject: [VOTE] Apache Yetus 0.7.0-RC1 @@ -274,8 +279,6 @@ Afterward, the artifacts should be visible via the web under the same URL used w [2]: https://www.apache.org/foundation/glossary.html#MajorityApproval [3]: to find this in your local timezone see: https://s.apache.org/yetus-0.7.0-rc1-close -That final short link should point to some online timezone conversion utility. ASF votes often use timeanddate.com's Event Time Announcer: http://www.timeanddate.com/worldclock/fixedform.html. - 1. Close the vote after the deadline. Once the deadline in the vote thread passes, tally the vote and post a suitable response that changes the subject line to start with "[RESULT]". If the vote failed, ensure there are issues in JIRA for any problems brought up. When they are closed, repeat the steps for creating a release candidate. If the vote passed, proceed to the [Cleanup section](#cleanup) ## Verification @@ -288,76 +291,89 @@ You will need to download the release candidate files, include the artifacts and For example, if we use the URL from our exemplar VOTE email, the process would look like this: - $ wget --recursive --no-parent --quiet 'https://dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/' - $ find dist.apache.org/ -type f - - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.asc - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.sha512 - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.mds - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/index.html - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.asc - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.sha512 - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.mds - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.asc - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.sha512 - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.mds - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.asc - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.sha512 - dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.mds - dist.apache.org//robots.txt +```bash +$ wget --recursive --no-parent --quiet 'https://dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/' +$ find dist.apache.org/ -type f + +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.asc +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.sha512 +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/CHANGELOG.md.mds +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/index.html +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.asc +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.sha512 +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/RELEASENOTES.md.mds +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.asc +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.sha512 +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-bin.tar.gz.mds +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.asc +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.sha512 +dist.apache.org//repos/dist/dev/yetus/0.7.0-RC1/apache-yetus-0.7.0-src.tar.gz.mds +dist.apache.org//robots.txt +``` Lastly, if you haven't verified a release before, you'll need to download and import the public keys for the project's release managers. The public keys are located in the KEYS file that should have been mentioned in the [VOTE] thread announcement. The specific output of the following commands will vary depending on how many release managers there have been and which keys, if any, you have previously imported. - $ curl --output KEYS.yetus --silent 'https://www.apache.org/dist/yetus/KEYS' - $ gpg --import KEYS.yetus - gpg: key 0D80DB7C: "Sean Busbey (CODE SIGNING KEY) " not changed - gpg: Total number processed: 1 - gpg: unchanged: 1 +```bash +$ curl --output KEYS.yetus --silent 'https://www.apache.org/dist/yetus/KEYS' +$ gpg --import KEYS.yetus +gpg: key 0D80DB7C: "Sean Busbey (CODE SIGNING KEY) " not changed +gpg: Total number processed: 1 +gpg: unchanged: 1 +``` ### ASF required checks ASF policies require that binding votes on releases be cast only after verifying proper licensing and provenance. For specific details, you should read the [ASF Release Policy's section entitled What Must Every ASF Release Contain?](https://www.apache.org/dev/release.html#what-must-every-release-contain) as well as the informational page [What We Sign](https://www.apache.org/info/verification.html). The following is a non-normative set of guidelines. -1. You MUST make sure each of the signatures matches. For example, using gpg and taking a fictional source artifact: +1. You MUST make sure each of the signatures matches. As noted in the informational page [What We Sign](https://www.apache.org/info/verification.html), if you don't have the signer's key in your web of trust the output of the verify command will point this out. You should refer to it for guidance. For example, using gpg and taking a fictional source artifact: - $ cd dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/ - $ gpg --verify apache-yetus-0.7.0-src.tar.gz.asc apache-yetus-0.7.0-src.tar.gz - gpg: Signature made Fri Dec 11 11:50:56 2015 CST using RSA key ID 0D80DB7C - gpg: Good signature from "Sean Busbey (CODE SIGNING KEY) " -As noted in the informational page [What We Sign](https://www.apache.org/info/verification.html), if you don't have the signer's key in your web of trust the output of the verify command will point this out. You should refer to it for guidance. + ```bash + $ cd dist.apache.org/repos/dist/dev/yetus/0.7.0-RC1/ + $ gpg --verify apache-yetus-0.7.0-src.tar.gz.asc apache-yetus-0.7.0-src.tar.gz + gpg: Signature made Fri Dec 11 11:50:56 2015 CST using RSA key ID 0D80DB7C + gpg: Good signature from "Sean Busbey (CODE SIGNING KEY) " + ``` 1. You MUST make sure the provided hashes match the provided artifact. - $ gpg --print-mds apache-yetus-0.7.0-src.tar.gz >apache-yetus-0.7.0-src.tar.gz.my_mds - $ diff apache-yetus-0.7.0-src.tar.gz.mds apache-yetus-0.7.0-src.tar.gz.my_mds - $ shasum -a 512 apache-yetus-0.7.0-src.tar.gz >apache-yetus-0.7.0-src.tar.gz.my_sha512 - $ diff apache-yetus-0.7.0-src.tar.gz.sha512 apache-yetus-0.7.0-src.tar.gz.my_sha512 + ```bash + $ gpg --print-mds apache-yetus-0.7.0-src.tar.gz >apache-yetus-0.7.0-src.tar.gz.my_mds + $ diff apache-yetus-0.7.0-src.tar.gz.mds apache-yetus-0.7.0-src.tar.gz.my_mds + $ shasum -a 512 apache-yetus-0.7.0-src.tar.gz >apache-yetus-0.7.0-src.tar.gz.my_sha512 + $ diff apache-yetus-0.7.0-src.tar.gz.sha512 apache-yetus-0.7.0-src.tar.gz.my_sha512 + ``` 1. You MUST make sure artifacts abide by the ASF Licensing Policy. You should read through [the ASF Licensing Policy](https://www.apache.org/legal/resolved), especially if your vote will be binding. As a quick guide: - * our software must be under the Apache Software License version 2.0 and this must be noted with a proper LICENSE and NOTICE file in each artifact that can hold them. - * our source code must meet the ASF policy on proper license notifications. Read the ASF Legal Committee's [Source Header Licensing Guide](https://apache.org/legal/src-headers.html) - * our LICENSE and NOTICE files must correctly propagate licensing information for bundled products. The [Foundation's Licensing HOWTO Guide](https://www.apache.org/dev/licensing-howto.html) provides guidance on how these files should be maintained. - * our software must only bundle compatibly licensed products; read [the Licensing Policy's Category A list for compatible licenses](https://www.apache.org/legal/resolved#category-a). - * our software may only have a runtime dependency on a product with a prohibit license if its use is optional; read [the Licensing Policy's Category X list for prohibited licenses](https://www.apache.org/legal/resolved#category-x) and [the Licensing Policy's explanation of optional runtime dependencies](https://www.apache.org/legal/resolved#optional). + - Our software must be under the Apache Software License version 2.0 and this must be noted with a proper `LICENSE` and `NOTICE` file in each artifact that can hold them. + - Our source code must meet the ASF policy on proper license notifications. Read the ASF Legal Committee's [Source Header Licensing Guide](https://apache.org/legal/src-headers.html) + - Our `LICENSE` and `NOTICE` files must correctly propagate licensing information for bundled products. The [Foundation's Licensing HOWTO Guide](https://www.apache.org/dev/licensing-howto.html) provides guidance on how these files should be maintained. + - Our software must only bundle compatibly licensed products; read [the Licensing Policy's Category A list for compatible licenses](https://www.apache.org/legal/resolved#category-a). + - Our software may only have a runtime dependency on a product with a prohibit license if its use is optional; read [the Licensing Policy's Category X list for prohibited licenses](https://www.apache.org/legal/resolved#category-x) and [the Licensing Policy's explanation of optional runtime dependencies](https://www.apache.org/legal/resolved#optional). 1. You SHOULD make sure the source release artifact corresponds to the referenced commit hash in the [VOTE] thread. (This ASF policy is currently in DRAFT status.) The release tag is how we'll provide long-term provenance information for our downstream users. Since the release's source code artifact will be the canonical representation of the release we vote on, it is essential that it matches the contents of the version control system's tag. Given our example above, you can check this w [...] - $ mkdir apache-yetus-0.7.0-src_unpack - $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz - $ git clone --single-branch --depth=1 --branch YETUS-585 'https://github.com/apache/yetus.git' apache-yetus-0.7.0-RC1-tag - $ diff -r apache-yetus-0.7.0-RC1-tag apache-yetus-0.7.0-src_unpack/yetus-project-0.7.0 - $ echo $? - 0 + ```bash + $ mkdir apache-yetus-0.7.0-src_unpack + $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz + $ git clone --single-branch --depth=1 --branch YETUS-585 `https://github.com/apache/yetus.git` apache-yetus-0.7.0-RC1-tag + $ diff -r apache-yetus-0.7.0-RC1-tag apache-yetus-0.7.0-src_unpack/yetus-project-0.7.0 + $ echo $? + 0 + ``` + 1. You MUST make sure any non-source artifacts can be derived from the source artifact. Since the source artifact is the canonical representation of our release, any other artifacts we distribute must be just for the convenience of our downstream users. As such, one must be able to derive them from the source artifact. Currently, you can generate all of the artifacts we distribute for convenience using the same commands used to create the release artifacts. - $ mkdir apache-yetus-0.7.0-src_unpack - $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz - $ cd apache-yetus-0.7.0-src_unpack/yetus-project-0.7.0 - $ mvn clean install -This will create a yetus-dist/target/ directory that contains the tarball binary distribution files. + ```bash + $ mkdir apache-yetus-0.7.0-src_unpack + $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz + $ cd apache-yetus-0.7.0-src_unpack/yetus-project-0.7.0 + $ mvn clean install + ``` + +This will create a `yetus-dist/target/` directory that contains the tarball binary distribution files. ### Community recommended checks @@ -365,133 +381,137 @@ If you've gone through all of the ASF required checks, you'll already have made 1. Test Precommit. The smart-apply-patch and test-patch scripts don't get flexed as a part of the above candidate verification. If you have a downstream project you regularly use, it should suffice to attempt local verification of a contribution. If that project happens to be an ASF project with an example personality, this should be as simple as finding an issue in patch-available status. - $ cd path/to/my/repo/for/hbase - $ /some/path/to/the/unpacked/candidate/bin/test-patch --project=hbase HBASE-1772 - ...SNIP... - -1 overall - - | Vote | Subsystem | Runtime | Comment - ============================================================================ - | 0 | reexec | 0m 0s | Docker mode activated. - | +1 | hbaseanti | 0m 0s | Patch does not have any anti-patterns. - | +1 | @\author | 0m 0s | The patch does not contain any @\author - | | | | tags. - | +1 | test4tests | 0m 0s | The patch appears to include 2 new or - | | | | modified test files. - | +1 | mvninstall | 4m 41s | master passed - | +1 | compile | 1m 4s | master passed with JDK v1.8.0_72 - | +1 | compile | 0m 57s | master passed with JDK v1.7.0_95 - | +1 | checkstyle | 0m 36s | master passed - | +1 | mvneclipse | 0m 35s | master passed - | -1 | findbugs | 1m 6s | hbase-client in master has 19 extant - | | | | Findbugs warnings. - | -1 | findbugs | 2m 8s | hbase-server in master has 84 extant - | | | | Findbugs warnings. - | -1 | javadoc | 0m 23s | hbase-client in master failed with JDK - | | | | v1.8.0_72. - | -1 | javadoc | 0m 34s | hbase-server in master failed with JDK - | | | | v1.8.0_72. - | +1 | javadoc | 0m 57s | master passed with JDK v1.7.0_95 - | +1 | mvninstall | 1m 3s | the patch passed - | +1 | compile | 0m 59s | the patch passed with JDK v1.8.0_72 - | +1 | javac | 0m 59s | the patch passed - | +1 | compile | 0m 59s | the patch passed with JDK v1.7.0_95 - | +1 | javac | 0m 59s | the patch passed - | +1 | checkstyle | 0m 32s | the patch passed - | +1 | mvneclipse | 0m 28s | the patch passed - | +1 | whitespace | 0m 0s | Patch has no whitespace issues. - | +1 | hadoopcheck | 4m 28s | Patch does not cause any errors with - | | | | Hadoop 2.4.1 2.5.2 2.6.0. - | +1 | findbugs | 3m 37s | the patch passed - | -1 | javadoc | 0m 24s | hbase-client in the patch failed with - | | | | JDK v1.8.0_72. - | -1 | javadoc | 0m 36s | hbase-server in the patch failed with - | | | | JDK v1.8.0_72. - | +1 | javadoc | 1m 2s | the patch passed with JDK v1.7.0_95 - | +1 | unit | 1m 23s | hbase-client in the patch passed with - | | | | JDK v1.8.0_72. - | -1 | unit | 67m 12s | hbase-server in the patch failed with - | | | | JDK v1.8.0_72. - | +1 | unit | 1m 28s | hbase-client in the patch passed with - | | | | JDK v1.7.0_95. - | -1 | unit | 66m 16s | hbase-server in the patch failed with - | | | | JDK v1.7.0_95. - | +1 | asflicense | 0m 30s | Patch does not generate ASF License - | | | | warnings. - | | | 177m 13s | - - - Reason | Tests - JDK v1.8.0_72 Failed junit tests | hadoop.hbase.client.TestMultiParallel - JDK v1.7.0_95 Failed junit tests | hadoop.hbase.client.TestMultiParallel - - - || Subsystem || Report/Notes || - ============================================================================ - | Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-11 | - | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12787466/HBASE-1772.patch | - | JIRA Issue | HBASE-15198 | - | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | - | uname | Linux 67e02eb9aeea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | - | Build tool | maven | - | Personality | /testptch/patchprocess/precommit/personality/hbase.sh | - | git revision | master / 81a6fff | - | findbugs | v2.0.1 | - | findbugs | /testptch/patchprocess/branch-findbugs-hbase-client-warnings.html | - | findbugs | /testptch/patchprocess/branch-findbugs-hbase-server-warnings.html | - | javadoc | /testptch/patchprocess/branch-javadoc-hbase-client-jdk1.8.0_72.txt | - | javadoc | /testptch/patchprocess/branch-javadoc-hbase-server-jdk1.8.0_72.txt | - | javadoc | /testptch/patchprocess/patch-javadoc-hbase-client-jdk1.8.0_72.txt | - | javadoc | /testptch/patchprocess/patch-javadoc-hbase-server-jdk1.8.0_72.txt | - | unit | /testptch/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt | - | unit | /testptch/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt | - | unit test logs | /testptch/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt /testptch/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt | - | modules | C: hbase-client hbase-server U: . | - | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | -1. Test Audience Annotations. If you have a downstream project that relies on the audience annotations project, you should be able to install the jars locally and test with the updated verison. - - $ mkdir apache-yetus-0.7.0-src_unpack - $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz - $ cd apache-yetus-0.7.0-src_unpack/yetus-0.7.0 - $ mvn --batch-mode install - ...SNIP... - [INFO] ------------------------------------------------------------------------ - [INFO] BUILD SUCCESS - [INFO] ------------------------------------------------------------------------ - [INFO] Total time: 3.539 s - [INFO] Finished at: 2016-02-13T02:12:39-06:00 - [INFO] Final Memory: 14M/160M - [INFO] ------------------------------------------------------------------------ - $ cd path/to/your/project - $ vim pom.xml # edit version to be e.g. 0.7.0 - $ mvn verify - ...SNIP... - [INFO] ------------------------------------------------------------------------ - [INFO] BUILD SUCCESS - [INFO] ------------------------------------------------------------------------ - [INFO] Total time: 7.539 m - [INFO] Finished at: 2016-02-13T02:13:39-06:00 - [INFO] Final Memory: 14M/160M - [INFO] ------------------------------------------------------------------------ + ```bash + $ cd path/to/my/repo/for/hbase + $ /some/path/to/the/unpacked/candidate/bin/test-patch --project=hbase HBASE-1772 + ...SNIP... + -1 overall + + | Vote | Subsystem | Runtime | Comment + ============================================================================ + | 0 | reexec | 0m 0s | Docker mode activated. + | +1 | hbaseanti | 0m 0s | Patch does not have any anti-patterns. + | +1 | @\author | 0m 0s | The patch does not contain any @\author + | | | | tags. + | +1 | test4tests | 0m 0s | The patch appears to include 2 new or + | | | | modified test files. + | +1 | mvninstall | 4m 41s | master passed + | +1 | compile | 1m 4s | master passed with JDK v1.8.0_72 + | +1 | compile | 0m 57s | master passed with JDK v1.7.0_95 + | +1 | checkstyle | 0m 36s | master passed + | +1 | mvneclipse | 0m 35s | master passed + | -1 | findbugs | 1m 6s | hbase-client in master has 19 extant + | | | | Findbugs warnings. + | -1 | findbugs | 2m 8s | hbase-server in master has 84 extant + | | | | Findbugs warnings. + | -1 | javadoc | 0m 23s | hbase-client in master failed with JDK + | | | | v1.8.0_72. + | -1 | javadoc | 0m 34s | hbase-server in master failed with JDK + | | | | v1.8.0_72. + | +1 | javadoc | 0m 57s | master passed with JDK v1.7.0_95 + | +1 | mvninstall | 1m 3s | the patch passed + | +1 | compile | 0m 59s | the patch passed with JDK v1.8.0_72 + | +1 | javac | 0m 59s | the patch passed + | +1 | compile | 0m 59s | the patch passed with JDK v1.7.0_95 + | +1 | javac | 0m 59s | the patch passed + | +1 | checkstyle | 0m 32s | the patch passed + | +1 | mvneclipse | 0m 28s | the patch passed + | +1 | whitespace | 0m 0s | Patch has no whitespace issues. + | +1 | hadoopcheck | 4m 28s | Patch does not cause any errors with + | | | | Hadoop 2.4.1 2.5.2 2.6.0. + | +1 | findbugs | 3m 37s | the patch passed + | -1 | javadoc | 0m 24s | hbase-client in the patch failed with + | | | | JDK v1.8.0_72. + | -1 | javadoc | 0m 36s | hbase-server in the patch failed with + | | | | JDK v1.8.0_72. + | +1 | javadoc | 1m 2s | the patch passed with JDK v1.7.0_95 + | +1 | unit | 1m 23s | hbase-client in the patch passed with + | | | | JDK v1.8.0_72. + | -1 | unit | 67m 12s | hbase-server in the patch failed with + | | | | JDK v1.8.0_72. + | +1 | unit | 1m 28s | hbase-client in the patch passed with + | | | | JDK v1.7.0_95. + | -1 | unit | 66m 16s | hbase-server in the patch failed with + | | | | JDK v1.7.0_95. + | +1 | asflicense | 0m 30s | Patch does not generate ASF License + | | | | warnings. + | | | 177m 13s | + + + Reason | Tests + JDK v1.8.0_72 Failed junit tests | hadoop.hbase.client.TestMultiParallel + JDK v1.7.0_95 Failed junit tests | hadoop.hbase.client.TestMultiParallel + + + || Subsystem || Report/Notes || + ============================================================================ + | Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-11 | + | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12787466/HBASE-1772.patch | + | JIRA Issue | HBASE-15198 | + | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | + | uname | Linux 67e02eb9aeea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | + | Build tool | maven | + | Personality | /testptch/patchprocess/precommit/personality/hbase.sh | + | git revision | master / 81a6fff | + | findbugs | v2.0.1 | + | findbugs | /testptch/patchprocess/branch-findbugs-hbase-client-warnings.html | + | findbugs | /testptch/patchprocess/branch-findbugs-hbase-server-warnings.html | + | javadoc | /testptch/patchprocess/branch-javadoc-hbase-client-jdk1.8.0_72.txt | + | javadoc | /testptch/patchprocess/branch-javadoc-hbase-server-jdk1.8.0_72.txt | + | javadoc | /testptch/patchprocess/patch-javadoc-hbase-client-jdk1.8.0_72.txt | + | javadoc | /testptch/patchprocess/patch-javadoc-hbase-server-jdk1.8.0_72.txt | + | unit | /testptch/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt | + | unit | /testptch/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt | + | unit test logs | /testptch/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt /testptch/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt | + | modules | C: hbase-client hbase-server U: . | + | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | + ``` + +1. Test Audience Annotations. If you have a downstream project that relies on the audience annotations project, you should be able to install the jars locally and test with the updated version. + + ```bash + $ mkdir apache-yetus-0.7.0-src_unpack + $ tar -C apache-yetus-0.7.0-src_unpack -xzf apache-yetus-0.7.0-src.tar.gz + $ cd apache-yetus-0.7.0-src_unpack/yetus-0.7.0 + $ mvn --batch-mode install + ...SNIP... + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + [INFO] Total time: 3.539 s + [INFO] Finished at: 2016-02-13T02:12:39-06:00 + [INFO] Final Memory: 14M/160M + [INFO] ------------------------------------------------------------------------ + $ cd path/to/your/project + $ vim pom.xml # edit version to be e.g. 0.7.0 + $ mvn verify + ...SNIP... + [INFO] ------------------------------------------------------------------------ + [INFO] BUILD SUCCESS + [INFO] ------------------------------------------------------------------------ + [INFO] Total time: 7.539 m + [INFO] Finished at: 2016-02-13T02:13:39-06:00 + [INFO] Final Memory: 14M/160M + [INFO] ------------------------------------------------------------------------ + ``` ## Cleanup Once a release candidate obtains majority approval from the PMC, there are several final maintenance tasks you must perform to close out the release. -1. Create shortcut links to the vote thread (e.g., https://s.apache.org/yetus-0.7.0-rc1-vote) and the result (e.g., https://s.apache.org/yetus-0.7.0-vote-passes) that point to the archives on mail-archives.apache.org. Be aware that it may take several hours for the archive to get the posts that need to be referenced. - +1. Create shortcut links to the vote thread (e.g., ) and the result (e.g., ) that point to the archives on mail-archives.apache.org. Be aware that it may take several hours for the archive to get the posts that need to be referenced. 1. Produce a signed release tag. You should create a signed tag and push it to the asf repo. The tag's message should include ASF-shortened links to the vote and results. It should be named 'rel/_version_' so that it will be immutable due to ASF infra's git configuration. Presuming we're working on the 0.7.0 release and the RC1 example above has passed: $ git config --global user.signingkey # if you've never configured $ git tag --sign rel/0.7.0 1e8f4588906a51317207092bd97b35687f2e3fa3 -Example commit message: + Example commit message: YETUS-XXX. tag Apache Yetus 0.7.0 release. vote thread: https://s.apache.org/yetus-0.7.0-rc1-vote results: https://s.apache.org/yetus-0.7.0-vote-passes -Then push: + Then push: $ git push origin rel/0.7.0 1. Add the release to the ASF reporter tool. To make our project reports for the ASF Board easier, you should include the release in the [Apache Committee Report Helper website](https://reporter.apache.org/addrelease.html?yetus). Be sure to use the date release artifacts first were pushed to the distribution area, which should be the same release date as in JIRA. Note that this website is only available to PMC members. If you are not yet in the PMC, please ask them to add the release inf [...] @@ -503,7 +523,7 @@ Then push: $ cp path/to/yetus-dist-dev/0.7.0-RC1/* 0.7.0 $ svn add 0.7.0 $ svn commit -m "Publish Apache Yetus 0.7.0" -It may take up to 24 hours for the artifacts to make their way to the various mirrors. You should not announce the release until after this period. + It may take up to 24 hours for the artifacts to make their way to the various mirrors. You should not announce the release until after this period. 1. Remove candidates from the staging area. Once you have moved the artifacts into the distribution area, they no longer need to be in the staging area and should be cleaned up as a courtesy to future release managers. $ svn co https://dist.apache.org/repos/dist/dev/yetus/ yetus-dist-dev @@ -534,32 +554,38 @@ It may take up to 24 hours for the artifacts to make their way to the various mi 1. Go to the [ASF repository](https://repository.apache.org/) and click 'Release' to put the RC Maven artifacts into the release repository. 1. Mark JIRA version as released. Browse to the [project version management page for the YETUS JIRA tracker](https://issues.apache.org/jira/plugins/servlet/project-config/YETUS/versions). Mouse over the version you are managing, click on the gear in the far right and select Release. 1. Delete staging branch. Now that there is an immutable tag for the release, all commits leading up to that release will be maintained by git. Should we need a future maintenance release after this version, we can reestablish the branch based off of the release tag. - $ git push origin :YETUS-XXX 1. Update the Mac OS X Homebrew Formula: - $ vim Formula/yetus.rb - $ # change URL point to the new version - $ # update the sha256. e.g., shasum -a 256 bin.gz - $ # test the formula: - $ brew install --build-from-source Formula/yetus.rb + ```bash + $ vim Formula/yetus.rb + $ # change URL point to the new version + $ # update the sha256. e.g., shasum -a 256 bin.gz + $ # test the formula: + $ brew install --build-from-source Formula/yetus.rb + ``` + 1. Update the documentation in the git master branch for the new release. Remove the oldest release and add the latest. - $ cd asf-site-src - $ # Add the release to the releases data file - $ vim data/versions.yml - $ vim pom.xml - $ # add the two stanzas - $ git add -p - $ git add asf-site-src/pom.xml - $ git commit -Example commit message: + ```bash + $ cd asf-site-src + $ # Add the release to the releases data file + $ vim data/versions.yml + $ vim pom.xml + $ # add the two stanzas + $ git add -p + $ git add asf-site-src/pom.xml + $ git commit + ``` - YETUS-XXX. add release 0.7.0. + - Example commit message: - - list in releases - - remove 0.4.0, add 0.7.0 to pom.xml + ```text + YETUS-XXX. add release 0.7.0. + - list in releases + - remove 0.4.0, add 0.7.0 to pom.xml + ``` 1. You should then post this patch for review. Once you've gotten feedback, it's fine to push the patch to the ASF source repo immediately so long as the updated website is not published. 1. Publish website updates. After the 24 hour window needed for the release artifacts to make their way to the variety of mirrors, you should render the website and publish it using the instructions found in [Maintaining the Yetus Website](../website). @@ -624,11 +650,11 @@ Example commit message: -- Meg Smith Apache Yetus PMC -If you'd like feedback on the draft, feel free to post it for review on your release issue. + If you'd like feedback on the draft, feel free to post it for review on your release issue. 1. Send announcement emails. After the 24 hour window needed for the release artifacts to make their way to the variety of mirrors, you should send the announcement email. The email should come from your apache.org email address and at a minimum should go to the dev@yetus.apache.org and announce@apache.org lists. For details see [the ASF Release Policy section How Should Releases Be Announced?](https://www.apache.org/dev/release.html#release-announcements). Additionally, you may want to [...] 1. Send tweet. Once the message to the ASF-wide announce list has made it to the public archive, you should draft a tweet with a link to the announcement. You should use the ASF link shortener and a descriptive name. For example, the 0.7.0 release could use Apache Yetus 0.7.0 has been released: https://s.apache.org/yetus-0.7.0-announce -This tweet should come from the official [@ApacheYetus](https://twitter.com/ApacheYetus/) account. Currently, only PMC members have access to it. If you are not yet on the PMC, please ask for the PMC to post the tweet once your email is available in the archives. + This tweet should come from the official [@ApacheYetus](https://twitter.com/ApacheYetus/) account. Currently, only PMC members have access to it. If you are not yet on the PMC, please ask for the PMC to post the tweet once your email is available in the archives. diff --git a/asf-site-src/source/contribute/website.md b/asf-site-src/source/contribute/website.md index c2f24a2..a5e0578 100644 --- a/asf-site-src/source/contribute/website.md +++ b/asf-site-src/source/contribute/website.md @@ -19,6 +19,16 @@ # Maintaining the Yetus Website + + +* [Make changes in asf-site-src/source](#make-changes-in-asf-site-srcsource) +* [Make changes to API Docs](#make-changes-to-api-docs) +* [Generating the website](#generating-the-website) +* [Live Development](#live-development) +* [Publishing the Site](#publishing-the-site) + + + We use [Middleman](https://middlemanapp.com/) to generate the website content from markdown and other dynamic templates.If you're interested in digging into how our site makes use of Middleman, or if you run into a problem, you should start by reading [Middleman's excellent documentation](https://middlemanapp.com/basics/install/). @@ -27,16 +37,18 @@ by reading [Middleman's excellent documentation](https://middlemanapp.com/basics NOTE: You MUST have run `mvn install` at least once prior to running `mvn site`. -The following steps assume you have a working ruby 2.x environment setup: +The following steps assume you have a working ruby 2.3+ environment setup: ```bash $ sudo gem install bundler $ cd asf-site-src $ bundle install ``` -and a working python 2.x environment for [releasedocmaker](../in-progress/releasedocmaker/). + +and a working Python 2.7 environment for [releasedocmaker](../in-progress/releasedocmaker/). ## Make changes in asf-site-src/source + Make any changes in the source directory: ```bash @@ -45,6 +57,7 @@ vi contribute.html.md ``` ## Make changes to API Docs + Optionally, you can update the generated API docs from other parts of the project. If they have been updated then the middleman build will pick up the changes. e.g. Precommit changes will be picked up by the Middleman build. @@ -55,6 +68,7 @@ vi 01-common.sh ``` ## Generating the website + To generate the static website for Apache Yetus run the following command at the root directory: ```bash @@ -69,6 +83,7 @@ open asf-site-src/target/site/index.html ``` ## Live Development + Live development of the site enables automatic reload when changes are saved. To enable, run the following commands and then open a browser and navigate to [http://localhost:4567](http://localhost:4567/) @@ -79,6 +94,7 @@ bundle exec middleman ``` ## Publishing the Site + Commit the publish directory to the asf-site branch. Presuming we start in a directory that holds your normal Yetus check out: ```bash diff --git a/asf-site-src/source/documentation/in-progress/interface-classification.md b/asf-site-src/source/documentation/in-progress/interface-classification.md index b40e8b3..6f1a9dc 100644 --- a/asf-site-src/source/documentation/in-progress/interface-classification.md +++ b/asf-site-src/source/documentation/in-progress/interface-classification.md @@ -12,38 +12,45 @@ limitations under the License. See accompanying LICENSE file. --> -Apache Yetus Interface Taxonomy: Audience and Stability Classification -================================================================ +# Apache Yetus Interface Taxonomy: Audience and Stability Classification -Motivation ----------- + -The interface taxonomy classification provided by Apache Yetus annotations is for guidance to -developers and users of interfaces. The classification guides a developer to -declare the targeted audience or users of an interface and also its stability. +* [Motivation](#motivation) +* [Interface Classification](#interface-classification) + * [Audience](#audience) + * [Private](#private) + * [Limited-Private](#limited-private) + * [Public](#public) + * [Stability](#stability) + * [Stable](#stable) + * [Evolving](#evolving) + * [Deprecated](#deprecated) +* [How are the Classifications Recorded](#how-are-the-classifications-recorded) +* [FAQ](#faq) + + + +# Motivation + +The interface taxonomy classification provided by Apache Yetus annotations is for guidance to developers and users of interfaces. The classification guides a developer to declare the targeted audience or users of an interface and also its stability. * Benefits to the user of an interface: Knows which interfaces to use or not use and their stability. * Benefits to the developer: to prevent accidental changes of interfaces and hence accidental impact on users or other components or system. This is - particularly useful in large systems with many developers who may not all have - a shared state/history of the project. + particularly useful in large systems with many developers who may not all have a shared state/history of the project. -Interface Classification ------------------------- +# Interface Classification Yetus provides the following interface classification, derived from the [OpenSolaris taxonomy](http://web.archive.org/web/20061013114610/http://opensolaris.org/os/community/arc/policies/interface-taxonomy/) and, to some extent, from taxonomy used inside Yahoo. Interfaces have two main attributes: Audience and Stability -### Audience +## Audience -Audience denotes the potential consumers of the interface. While many interfaces -are internal/private to the implementation, others are public/external interfaces -and are meant for wider consumption by applications and/or clients. For example, -POSIX definitions in libc are external, while large parts of the kernel are internal or private interfaces. -Also, some interfaces are targeted towards other specific subsystems. +Audience denotes the potential consumers of the interface. While many interfaces are internal/private to the implementation, others are public/external interfaces and are meant for wider consumption by applications and/or clients. For example, POSIX definitions in libc are external, while large parts of the kernel are internal or private interfaces. Also, some interfaces are targeted towards other specific subsystems. Identifying the audience of an interface helps define the impact of breaking it. For instance, it might be okay to break the compatibility of an interface @@ -53,38 +60,36 @@ users depend on. Yetus uses the following kinds of audience in order of increasing/wider visibility: -#### Private +### Private The interface is for internal use within a project(such as Apache Hadoop) and should not be used by applications or by other projects. It is subject to -change at anytime without notice. Most interfaces of a project are Private (also -referred to as project-private). +change at anytime without notice. Most interfaces of a project are Private (also referred to as project-private). -#### Limited-Private +### Limited-Private The interface is used by a specified set of projects or systems (typically closely related projects). Other projects or systems should not use the interface. Changes to the interface will be communicated/ negotiated with the -specified projects. For example, in the Apache Hadoop project, some interfaces are -LimitedPrivate{HDFS, MapReduce} in that they are private to the HDFS and +specified projects. For example, in the Apache Hadoop project, some interfaces are LimitedPrivate{HDFS, MapReduce} in that they are private to the HDFS and MapReduce subprojects. -#### Public +### Public The interface is for general use by any application. -### Stability +## Stability Stability denotes how stable an interface is, as in when incompatible changes to the interface are allowed. Yetus provides the following levels of stability. -#### Stable +### Stable Can evolve while retaining compatibility for minor release boundaries; in other words, incompatible changes to APIs marked Stable are generally only allowed at major releases (i.e. at m.0). -#### Evolving +### Evolving Evolving, but incompatible changes are allowed at minor release (i.e. m .x) @@ -100,111 +105,90 @@ as Not-an-interface is probably more appropriate than "Unstable". Examples of publicly visible interfaces that are unstable (i.e. not-an-interface): GUI, CLIs whose output format will change -#### Deprecated +### Deprecated APIs that could potentially be removed in the future and should not be used. -How are the Classifications Recorded -------------------------------------- - +# How are the Classifications Recorded [//]: # (This section needs improvement. Refer YETUS-458) - How should the classification be recorded for the annotated APIs? * Each interface or class will have the audience and stability recorded using annotations in org.apache.yetus.classification package. - * The javadoc generated by the maven target javadoc:javadoc lists only the public API. - -* One can derive the audience of java classes and java interfaces by the +* One can derive the audience of Java classes and Java interfaces by the audience of the package in which they are contained. Hence it is useful to - declare the audience of each java package as public or private (along with the - private audience variations). - -FAQ ---- - -* Why aren't the java scopes (private, package private and public) good enough? - * Java's scoping is not very complete. One is often forced to make a class - public in order for other internal components to use it. It does not have - friends or sub-package-private like C++. - -* But I can easily access a private implementation interface if it is Java public. - Where is the protection and control? - * The purpose of this is not providing absolute access control. Its purpose - is to communicate to users and developers. One can access private - implementation functions in libc; however if they change the internal - implementation details, your application will break and you will have - little sympathy from the folks who are supplying libc. If you use a - non-public interface you understand the risks. - + declare the audience of each Java package as public or private (along with the private audience variations). + +# FAQ + +* Why aren't the Java scopes (private, package private and public) good enough? + * Java's scoping is not very complete. One is often forced to make a class public in order for other internal components to use it. It does not have friends or sub-package-private like C++. +* But I can easily access a private implementation interface if it is Java public. Where is the protection and control? + * The purpose of this is not providing absolute access control. Its purpose + is to communicate to users and developers. One can access private + implementation functions in libc; however if they change the internal + implementation details, your application will break and you will have + little sympathy from the folks who are supplying libc. If you use a + non-public interface you understand the risks. * Why bother declaring the stability of a private interface? Aren't private interfaces always unstable? - * Private interfaces are not always unstable. In the cases where they are - stable they capture internal properties of the system and can communicate - these properties to its internal users and to developers of the interface. - * e.g. In HDFS, NN-DN protocol is private but stable and can help - implement rolling upgrades. It communicates that this interface should - not be changed in incompatible ways even though it is private. - * e.g. In HDFS, FSImage stability can help provide more flexible roll backs. - -* What is the harm in applications using a private interface that is stable? How - is it different than a public stable interface? - * While a private interface marked as stable is targeted to change only at - major releases, it may break at other times if the providers of that - interface are willing to changes the internal users of that - interface. Further, a public stable interface is less likely to break even - at major releases (even though it is allowed to break compatibility) - because the impact of the change is larger. If you use a private interface - (regardless of its stability) you run the risk of incompatibility. - -* Why bother with Limited-private? Isn't it giving special treatment to some projects? - That is not fair. - * First, most interfaces should be public or private; actually let us state - it even stronger: make it private unless you really want to expose it to - public for general use. - * Limited-private is for interfaces that are not intended for general - use. They are exposed to related projects that need special hooks. Such a - classification has a cost to both the supplier and consumer of the limited - interface. Both will have to work together if ever there is a need to - break the interface in the future; for example the supplier and the - consumers will have to work together to get coordinated releases of their - respective projects. This should not be taken lightly - if you can get - away with private then do so; if the interface is really for general use - for all applications then you should consider making it public. But remember - that making an interface public has huge responsibility. Sometimes - Limited-private is just right. - * A good example of a limited-private interface is BlockLocations in the Apache - Hadoop Project, This is fairly low-level interface that they are willing to - expose to MR and perhaps HBase. They are likely to change it down the road - and at that time they will have to get a coordinated effort with the MR - team to release matching releases. While MR and HDFS are always released - in sync today, they may change down the road. - * If you have a limited-private interface with many projects listed then you - are fooling yourself. It is practically public. - * It might be worth declaring a special audience classification called - {YourProjectName}-Private for your closely related projects. - -* Can't a private interface be treated as project-private also? For example what is - the harm in projects in the Apache Hadoop extended ecosystem, having access to - private classes? - * Do we want MR accessing class files that are implementation details inside - HDFS? There used to be many such layer violations in the Apache Hadoop - project that they have been cleaning up over the last few years. It is highly - undesirable for such layer violations to creep back in by no separation - between the major components like HDFS and MR. - + * Private interfaces are not always unstable. In the cases where they are + stable they capture internal properties of the system and can communicate + these properties to its internal users and to developers of the interface. + * e.g. In HDFS, NN-DN protocol is private but stable and can help + implement rolling upgrades. It communicates that this interface should + not be changed in incompatible ways even though it is private. + * e.g. In HDFS, FSImage stability can help provide more flexible roll backs. +* What is the harm in applications using a private interface that is stable? How is it different than a public stable interface? + * While a private interface marked as stable is targeted to change only at + major releases, it may break at other times if the providers of that + interface are willing to changes the internal users of that + interface. Further, a public stable interface is less likely to break even + at major releases (even though it is allowed to break compatibility) + because the impact of the change is larger. If you use a private interface + (regardless of its stability) you run the risk of incompatibility. +* Why bother with Limited-private? Isn't it giving special treatment to some projects? That is not fair. + * First, most interfaces should be public or private; actually let us state + it even stronger: make it private unless you really want to expose it to + public for general use. + * Limited-private is for interfaces that are not intended for general + use. They are exposed to related projects that need special hooks. Such a + classification has a cost to both the supplier and consumer of the limited + interface. Both will have to work together if ever there is a need to + break the interface in the future; for example the supplier and the + consumers will have to work together to get coordinated releases of their + respective projects. This should not be taken lightly - if you can get + away with private then do so; if the interface is really for general use + for all applications then you should consider making it public. But remember + that making an interface public has huge responsibility. Sometimes + Limited-private is just right. + * A good example of a limited-private interface is BlockLocations in the Apache + Hadoop Project, This is fairly low-level interface that they are willing to + expose to MR and perhaps HBase. They are likely to change it down the road + and at that time they will have to get a coordinated effort with the MR + team to release matching releases. While MR and HDFS are always released + in sync today, they may change down the road. + * If you have a limited-private interface with many projects listed then you are fooling yourself. It is practically public. + * It might be worth declaring a special audience classification called + {YourProjectName}-Private for your closely related projects. +* Can't a private interface be treated as project-private also? For example what is the harm in projects in the Apache Hadoop extended ecosystem, having access to private classes? + * Do we want MR accessing class files that are implementation details inside + HDFS? There used to be many such layer violations in the Apache Hadoop + project that they have been cleaning up over the last few years. It is highly + undesirable for such layer violations to creep back in by no separation + between the major components like HDFS and MR. * Aren't all public interfaces stable? - * One may mark a public interface as evolving in its early days. Here one is - promising to make an effort to make compatible changes but may need to - break it at minor releases. - * One example of a public interface that is unstable is where one is - providing an implementation of a standards-body based interface that is - still under development. For example, many companies, in an attempt to be - first to market, have provided implementations of a new NFS protocol even - when the protocol was not fully completed by IETF. The implementor cannot - evolve the interface in a fashion that causes least disruption because - the stability is controlled by the standards body. Hence it is appropriate - to label the interface as unstable. + * One may mark a public interface as evolving in its early days. Here one is + promising to make an effort to make compatible changes but may need to + break it at minor releases. + * One example of a public interface that is unstable is where one is + providing an implementation of a standards-body based interface that is + still under development. For example, many companies, in an attempt to be + first to market, have provided implementations of a new NFS protocol even + when the protocol was not fully completed by IETF. The implementor cannot + evolve the interface in a fashion that causes least disruption because + the stability is controlled by the standards body. Hence it is appropriate + to label the interface as unstable. diff --git a/asf-site-src/source/documentation/in-progress/precommit-admin.md b/asf-site-src/source/documentation/in-progress/precommit-admin.md index 49f13eb..2ce844a 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-admin.md +++ b/asf-site-src/source/documentation/in-progress/precommit-admin.md @@ -17,8 +17,14 @@ under the License. --> -Overview -======== +# Overview + + + +* [Project-Specifc Builds](#project-specifc-builds) +* [Jenkins Job Tokens](#jenkins-job-tokens) + + The jenkins-admin script is an automated way to submit JIRA issues to the Apache Yetus precommit testing framework. It works by: @@ -40,14 +46,13 @@ Additionally, the URL of the Jenkins server is expected to be in the `JENKINS_UR The very first run of the job should be done with the `--initialize` parameter to create the first `patch_tested.txt` file. Otherwise, the job will fail because a previous version of it cannot be downloaded from previous runs. -Project-Specifc Builds -======================= +# Project-Specifc Builds New builds are started via buildWithParameters call. Three parameters are added to the URL: - * token = Jenkins security token (see below) - * ISSUE\_NUM = JIRA issue number - * ATTACHMENT\_ID = JIRA attachment id +* token = Jenkins security token (see below) +* ISSUE\_NUM = JIRA issue number +* ATTACHMENT\_ID = JIRA attachment id By default, the Jenkins job name is expected to be `PreCommit-{project}`, where the project name matches the JIRA project name. Using the JIRA issue YETUS-1 with an attachment number of 2345 would result in the following URL: @@ -55,7 +60,9 @@ New builds are started via buildWithParameters call. Three parameters are added The `{JENKINS_URL}` can be overridden on a per project basis using the `--jenkins-url-override` option. This parameter allows for one job on one Jenkins server to direct different projects to different Jenkins servers. For example: +```bash jenkins-admin --jenkins-url-override=PROJ1=https://example.com/1 --jenkins-url-override=PROJ2=https://example.com/1 +``` would send all PROJ1 Jenkins jobs to the first URL and all PROJ2 jobs to the second URL. The `--jenkins-url-override` option may be listed as many times as necessary. @@ -63,21 +70,20 @@ The job name can be overridden via the `--jenkins-project-template` option. For .../job/PreCommit-YETUS-Build/buildwithParameters?... -Jenkins Job Tokens -================== +# Jenkins Job Tokens Currently, jenkins-admin supports the usage of Jenkins tokens for authentication via the `--jenkins-token` option. This option provides two ways to do tokens - * Flat tokens - * Template tokens +* Flat tokens +* Template tokens Flat tokens are a simple string. For example, `--jenkins-tokens=yetus` would require the `yetus` string to be listed as the token in the Jenkins job configuration that is being requested. On the other hand, template tokens perform some simple string substitution before being used. This exchange includes: - * {project} for the JIRA project name - * {issue} for the JIRA issue number - * {attachment} for the JIRA issue attachment id +* {project} for the JIRA project name +* {issue} for the JIRA issue number +* {attachment} for the JIRA issue attachment id For example, if JIRA issue YETUS-1 has an attachment of 2345, then `{project}` becomes `YETUS`, `{issue}` becomes `1`, and `{attachment}` becomes 2345. diff --git a/asf-site-src/source/documentation/in-progress/precommit-advanced.md b/asf-site-src/source/documentation/in-progress/precommit-advanced.md index 98a8fd9..4e5a68e 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-advanced.md +++ b/asf-site-src/source/documentation/in-progress/precommit-advanced.md @@ -17,14 +17,25 @@ under the License. --> -test-patch -========== +# Advanced Precommit -* [Process Reaper](#test-reaper) + + +* [Process Reaper](#process-reaper) * [Plug-ins](#plug-ins) + * [Common Plug-in Functions](#common-plug-in-functions) + * [Plug-in Importation](#plug-in-importation) + * [Test Plug-ins](#test-plug-ins) * [Personalities](#personalities) + * [Configuring for Other Projects](#configuring-for-other-projects) + * [Global Definitions](#global-definitions) + * [Test Determination](#test-determination) + * [Module & Profile Determination](#module--profile-determination) + * [Enabling Plug-ins](#enabling-plug-ins) * [Important Variables](#important-variables) + + # Process Reaper A common problem is the 'stuck' unit test. If bash v4.0 or higher is in use, Apache Yetus may be told to turn on the process reaper functionality. Using the `--reapearmode` option, this feature may be configured to either report and even kill left over processes that match provided regular expressions. @@ -58,32 +69,31 @@ Similarly, there are other functions that may be defined during the test-patch r HINT: It is recommended to make the pluginname relatively small, 10 characters at the most. Otherwise, the ASCII output table may be skewed. * pluginname\_usage - - executed when the help message is displayed. This is used to display the plug-in specific options for the user. + * executed when the help message is displayed. This is used to display the plug-in specific options for the user. * pluginname\_parse\_args - - executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment. + * executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment. * pluginname\_initialize - - After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc. + * After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc. * pluginname\_docker\_support - - Perform any necessary setup to configure Docker support for the given plugin. Typically this means adding parameters to the docker run command line via adding to the DOCKER\_EXTRAARGS array. + * Perform any necessary setup to configure Docker support for the given plugin. Typically this means adding parameters to the docker run command line via adding to the DOCKER\_EXTRAARGS array. * pluginname\_precheck - - executed prior to the patch being applied but after the git repository is setup. Returning a fail status here will exit test-patch. + * executed prior to the patch being applied but after the git repository is setup. Returning a fail status here will exit test-patch. * pluginname\_patchfile - - executed prior to the patch being applied but after the git repository is setup. This step is intended to perform tests on the content of the patch itself. + * executed prior to the patch being applied but after the git repository is setup. This step is intended to perform tests on the content of the patch itself. * pluginname\_precompile - - executed prior to the compilation part of the lifecycle. This is useful for doing setup work required by the compilation process. + * executed prior to the compilation part of the lifecycle. This is useful for doing setup work required by the compilation process. * pluginname\_postcompile - - This step happens after the compile phase. + * This step happens after the compile phase. * pluginname\_rebuild - - Any non-unit tests that require the source to be rebuilt in a destructive way should be run here. - + * Any non-unit tests that require the source to be rebuilt in a destructive way should be run here. ## Plug-in Importation @@ -101,29 +111,27 @@ If the `--skip-system-plugins` flag is passed, then only core.d is imported. Plug-ins geared towards independent tests are registered via: - ```bash add_test_type ``` -+ pluginname\_filefilter - - executed while determining which files trigger which tests. This function should use `add_test pluginname` to add the plug-in to the test list. +* pluginname\_filefilter + * executed while determining which files trigger which tests. This function should use `add_test pluginname` to add the plug-in to the test list. * pluginname\_compile - - executed immediately after the actual compilation. This step is intended to be used to verify the results and add extra checking of the compile phase and it's stdout/stderr. + * executed immediately after the actual compilation. This step is intended to be used to verify the results and add extra checking of the compile phase and it's stdout/stderr. * pluginname\_tests - - executed after the unit tests have completed. + * executed after the unit tests have completed. * pluginname\_clean - - executed to allow the plugin to remove all files that have been generate by this plugin. + * executed to allow the plugin to remove all files that have been generate by this plugin. * pluginname\_logfilter - - This functions should filter all lines relevant to this test from the logfile. It is called in preparation for the `calcdiffs` function. + * This functions should filter all lines relevant to this test from the logfile. It is called in preparation for the `calcdiffs` function. * pluginname\_calcdiffs - - - This allows for custom log file difference calculation used to determine the before and after views. The default is to use the last column of a colon delimited line of output and perform a diff. If the plug-in does not provide enough context, this may result in error skew. For example, if three lines in a row have "Missing period." as the error, test-patch will not be able to determine exactly which line caused the error. Plug-ins that have this issue will want to use this or po [...] + * This allows for custom log file difference calculation used to determine the before and after views. The default is to use the last column of a colon delimited line of output and perform a diff. If the plug-in does not provide enough context, this may result in error skew. For example, if three lines in a row have "Missing period." as the error, test-patch will not be able to determine exactly which line caused the error. Plug-ins that have this issue will want to use this or pote [...] NOTE: If the plug-in has support for maven, the maven_add_install `pluginname` should be executed. See more information in Custom Maven Tests in the build tool documentation. diff --git a/asf-site-src/source/documentation/in-progress/precommit-basic.md b/asf-site-src/source/documentation/in-progress/precommit-basic.md index 880a388..67a6b3a 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-basic.md +++ b/asf-site-src/source/documentation/in-progress/precommit-basic.md @@ -17,8 +17,7 @@ under the License. --> -test-patch -========== +# Basic Precommit @@ -114,15 +113,16 @@ Language Support, Licensing, and more: * [Apache Creadur Rat](http://creadur.apache.org/rat/) entries in build system * [checkstyle](http://checkstyle.sourceforge.net/) entries in build system (ant and maven only) * [FindBugs](http://findbugs.sourceforge.net/) entries in build system and 3.x executables - - NOTE: only one of FindBugs or SpotBugs may be used at a time. -* [SpotBugs](https://spotbugs.github.io/)) entries in build system and 3.x executables - - NOTE: only one of FindBugs or SpotBugs may be used at a time. + * NOTE: only one of FindBugs or SpotBugs may be used at a time. * [jshint](https://jshint.com) installed * [hadolint](https://github.com/hadolint/hadolint) installed +* [markdownlint-cli](https://github.com/igorshubovych/markdownlint-cli) installed * [Perl::Critic](http://perlcritic.com/) installed * [pylint](http://www.pylint.org/) installed * [rubocop](http://batsov.com/rubocop/) installed * [shellcheck](https://github.com/koalaman/shellcheck) installed, preferably 0.3.6 or higher +*[SpotBugs](https://spotbugs.github.io/)) entries in build system and 3.x executables + * NOTE: only one of FindBugs or SpotBugs may be used at a time. * [yamllint](https://github.com/adrienverge/yamllint) installed # First Steps @@ -162,7 +162,7 @@ $ test-patch --list-plugins You should see output similar to this: -``` +```text BUILDTOOLS: ant autoconf cmake gradle make maven nobuild TESTTYPES: @@ -321,7 +321,6 @@ $ test-patch --plugins=all https://gitlab.com/_a__w_/yetus/merge_requests/3.patc ... will process MR #3 on the \_a\_\_w\_/yetus repo. - ## Generic URLs Luckily, `test-patch` supports ways to provide unified diffs via URLs. @@ -366,7 +365,6 @@ $ test-patch --plugins=all --proclimit=10000 NOTE: The actual implementation of this feature is dependent upon the version of Bash. For bash v4 and higher (most operating systems), the fork bomb protection is generally only used for the build and QA tools. This means Apache Yetus should continue to function. For earlier versions of bash (e.g., OS X), the limit is applied to all of test-patch. If the limit is hit, Apache Yetus will itself likely crash. - # MultiJDK For many projects, it is useful to test Java code against multiple versions of JDKs at the same time. In combination with the `java` plug-in, `test-patch` can do this with the `--multijdkdirs` option: @@ -383,7 +381,7 @@ NOTE: JAVA\_HOME is always appended to the list of JDKs in MultiJDK mode. If JA `test-patch` also has a built-in mode (i.e., no plug-in required) to utilize Docker: -```bash +```diff <<<<<<< HEAD $ test-patch (other options) --docker ======= @@ -393,7 +391,6 @@ $ test-patch --docker This command will do some preliminary setup and then re-execute itself inside a Docker container. For more information on how to provide a custom Dockerfile and other Docker-specific features, see the specific [precommit Docker support](../precommit-docker) page and the [Apache Yetus Docker Hub Images](/yetus-docker-image) page for more information on the convenience Docker images. - # In Closing `test-patch` has many other features and command line options for the basic user. Many of these are self-explanatory. To see the list of options, run `test-patch` without any options or with `--help`. diff --git a/asf-site-src/source/documentation/in-progress/precommit-bugsystems.md b/asf-site-src/source/documentation/in-progress/precommit-bugsystems.md index d1f3dcc..ea49e5d 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-bugsystems.md +++ b/asf-site-src/source/documentation/in-progress/precommit-bugsystems.md @@ -17,8 +17,16 @@ under the License. --> -Bug System Support -================== +# Bug System Support + + + +* [Bugzilla Specific](#bugzilla-specific) +* [GitHub Specific](#github-specific) +* [GitLab Specific](#gitlab-specific) +* [JIRA Specific](#jira-specific) + + test-patch has the ability to support multiple bug systems. Bug tools have some extra hooks to fetch patches, line-level reporting, and posting a final report. Every bug system plug-in must have one line in order to be recognized: @@ -27,28 +35,22 @@ add_bugsystem ``` * pluginname\_locate\_patch - - - Given input from the user, download the patch if possible. + * Given input from the user, download the patch if possible. * pluginname\_determine\_branch - - - Using any heuristics available, return the branch to process, if possible. + * Using any heuristics available, return the branch to process, if possible. * pluginname\_determine\_issue - - - Using any heuristics available, set the issue, bug number, etc, for this bug system, if possible. This is typically used to fill in supplementary information in the final output table. + * Using any heuristics available, set the issue, bug number, etc, for this bug system, if possible. This is typically used to fill in supplementary information in the final output table. * pluginname\_write\_comment - - - Given text input, write this output to the bug system as a comment. NOTE: It is the bug system's responsibility to format appropriately. + * Given text input, write this output to the bug system as a comment. NOTE: It is the bug system's responsibility to format appropriately. * pluginname\_linecomments - - - This function allows for the system to write specific comments on specific lines if the bug system supports code review comments. + * This function allows for the system to write specific comments on specific lines if the bug system supports code review comments. * pluginname\_finalreport - - - Write the final result table to the bug system. + * Write the final result table to the bug system. # Bugzilla Specific @@ -60,13 +62,13 @@ $ test-patch (other options) BZ:4 ... will pull down Bugzilla ID #4. -Using the `--bugzilla-base-url` on the command line or BUGZILLA\_BASE\_URL in a project's personality will define the location of the Bugzilla instance. By default, it is https://bz.apache.org/bugzilla . +Using the `--bugzilla-base-url` on the command line or BUGZILLA\_BASE\_URL in a project's personality will define the location of the Bugzilla instance. By default, it is . # GitHub Specific GitHub supports the full range of functionality, including putting comments on individual lines. Be aware, however, that `test-patch` will (generally) require that GitHub PRs be fully squashed and rebased (i.e., a single commit) in many circumstances. -By default, the GitHub plug-in assumes that https://github.com is the base URL for GitHub. Enterprise users may override this with the `--github-base-url` for the normal web user interface and `--github-api-url` for the API URL. Personalities may use GITHUB\_API\_URL and GITHUB\_BASE\_URL. +By default, the GitHub plug-in assumes that is the base URL for GitHub. Enterprise users may override this with the `--github-base-url` for the normal web user interface and `--github-api-url` for the API URL. Personalities may use GITHUB\_API\_URL and GITHUB\_BASE\_URL. The specific repository on GitHub is defined with either `--github-repo` on the command line or GITHUB\_REPO in a personality. It should take the form of "user/repo". @@ -81,8 +83,8 @@ Both username and password options must be provided. GitHub pull requests may be directly processed on the command line in two ways: - * GH:(PR number) - * GHSHA:(PR SHA1 number) +* GH:(PR number) +* GHSHA:(PR SHA1 number) The GitHub bugsystem plugin will attempt to download the unified diff that the pull request references. Pull requests that are made off of a specific branch will switch the test repo to that branch, if permitted. If the pull request references a JIRA issue that matches the given JIRA issue regexp in the Subject, the JIRA plug-in will also be invoked as needed. @@ -91,17 +93,16 @@ Pull requests that are made off of a specific branch will switch the test repo t GitLab supports the full range of functionality, including putting comments on individual lines. Be aware, however, that `test-patch` will (generally) require that GitLab MRs be fully squashed and rebased (i.e., a single commit) in many circumstances. -By default, the GitLab plug-in assumes that https://gitlab.com is the base URL for GitHub. Enterprise users may override this with the `--gitlab-base-url` for the normal web user interface and `--gitlab-api-url` for the API URL. Personalities may use GITLAB\_API\_URL and GITLAB\_BASE\_URL. +By default, the GitLab plug-in assumes that is the base URL for GitLab. Enterprise users may override this with the `--gitlab-base-url` for the normal web user interface and `--gitlab-api-url` for the API URL. Personalities may use GITLAB\_API\_URL and GITLAB\_BASE\_URL. The specific repository on GitLab is defined with either `--gitlab-repo` on the command line or GITLAB\_REPO in a personality. It should take the form of "user/repo". In order to comment on issues or, depending upon the security setup of the repo, authentication credentials. The GitLab plug-in supports tokens via the `--gitlab-token` option or GITLAB\_TOKEN environment variable. - GitLab merge requests may be directly processed on the command line in two ways: - * GL:(MR number) - * GLSHA:(MR SHA1 number) +* GL:(MR number) +* GLSHA:(MR SHA1 number) The GitLab bugsystem plugin will attempt to download the unified diff that the merge request references. Merge requests that are made off of a specific branch will switch the test repo to that branch, if permitted. If the merge request references a JIRA issue that matches the given JIRA issue regexp in the Subject, the JIRA plug-in will also be invoked as needed. @@ -110,7 +111,7 @@ Merge requests that are made off of a specific branch will switch the test repo JIRA support allows both patch downloads and summary writes. It also supports branch detection-based upon the name of the attached patch file. -JIRA issues are invoked by matching the command line option to a specific regular expression as given by the `--jira-issue-re` option or via the JIRA\_ISSUE\_RE personality variable. By default, the plug-in uses https://issues.apache.org/jira as the JIRA instance to use. However that may be overwritten via the `--jira-base-url` option or personalities may define via JIRA\_URL. +JIRA issues are invoked by matching the command line option to a specific regular expression as given by the `--jira-issue-re` option or via the JIRA\_ISSUE\_RE personality variable. By default, the plug-in uses as the JIRA instance to use. However that may be overwritten via the `--jira-base-url` option or personalities may define via JIRA\_URL. In order to write information on the issue, JIRA requires username and password authentication using the `--jira-user`/`--jira-password` options or the JIRA\_USER and JIRA\_PASSWORD variables in a personality. diff --git a/asf-site-src/source/documentation/in-progress/precommit-buildtools.md b/asf-site-src/source/documentation/in-progress/precommit-buildtools.md index f817073..73911a4 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-buildtools.md +++ b/asf-site-src/source/documentation/in-progress/precommit-buildtools.md @@ -17,28 +17,38 @@ under the License. --> - - -Build Tool Support -=================== +# Build Tool Support * [Global Variables](#global-variables) + * [BUILDTOOLCWD](#buildtoolcwd) + * [UNSUPPORTED\_TEST](#unsupported_test) * [Required Functions](#required-functions) + * [pluginname\_buildfile](#pluginname_buildfile) + * [pluginname\_executor](#pluginname_executor) + * [pluginname\_modules\_worker](#pluginnamemodulesworker) + * [pluginname\_builtin_personality\_modules](#pluginnamebuiltin_personalitymodules) + * [pluginname\_builtin_personality\_file\_tests](#pluginnamebuiltin_personality_filetests) * [Optional Functions](#optional-functions) -* [Ant Specific](#ant-specific) - * [Command Arguments](#command-arguments) - * [Docker Mode](#docker-mode) + * [pluginname\_parse\_args](#pluginnameparseargs) + * [pluginname\_initialize](#pluginname_initialize) + * [pluginname\_reorder\_modules](#pluginnamereordermodules) + * [pluginname\_\(test\)\_logfilter](#pluginnametestlogfilter) + * [pluginname\_\(test\)_calcdiffs](#pluginnametestcalcdiffs) + * [pluginname\_docker\_support](#pluginnamedockersupport) +* [Apache Ant Specific](#apache-ant-specific) + * [Ant Command Arguments](#ant-command-arguments) + * [Ant in Docker Mode](#ant-in-docker-mode) * [autoconf Specific](#autoconf-specific) - * [Command Arguments](#command-arguments-1) + * [autoconf Command Arguments](#autoconf-command-arguments) * [CMAKE Specific](#cmake-specific) * [Gradle Specific](#gradle-specific) * [Make Specific](#make-specific) -* [Maven Specific](#maven-specific) - * [Command Arguments](#command-arguments-2) +* [Apache Maven Specific](#apache-maven-specific) + * [Command Arguments](#command-arguments) * [Per-instance Repositories](#per-instance-repositories) - * [Docker Mode](#docker-mode-1) + * [Maven in Docker Mode](#maven-in-docker-mode) * [Test Profile](#test-profile) * [Custom Maven Tests](#custom-maven-tests) @@ -52,87 +62,79 @@ add_build_tool # Global Variables -* BUILDTOOLCWD - - - This variable determines where the build tool's command (as returned by pluginname\_executor) should actually execute. It should be one of three values: - - * basedir - always execute at the root of the source tree - * module - switch to the directory as given by the module being processed - * /(path) - change to the directory as given by this absolute path. If the path does not exist, it will be created. +## BUILDTOOLCWD - If /(path) is used, two special substitutions may be made: +This variable determines where the build tool's command (as returned by pluginname\_executor) should actually execute. It should be one of three values: +* basedir - always execute at the root of the source tree +* module - (default) switch to the directory as given by the module being processed +* /(path) - change to the directory as given by this absolute path. This allows for custom directories to be created and used as necessary. If the path does not exist, it will be created. If /(path) is used, two special substitutions may be made: * @@@BASEDIR@@@ will be replaced with the root of the source tree * @@@MODULEDIR@@@ will be replaced with the module name - This allows for custom directories to be created and used as necessary. +## UNSUPPORTED\_TEST - - The default is module. - -* UNSUPPORTED\_TEST - - - If pluginname\_modules\_worker is given a test type that is not supported by the build system, set UNSUPPORTED\_TEST=true. If it is supported, set UNSUPPORTED\_TEST=false. +If pluginname\_modules\_worker is given a test type that is not supported by the build system, set UNSUPPORTED\_TEST=true. If it is supported, set UNSUPPORTED\_TEST=false. For example, the gradle build tool does not have a standard way to execute checkstyle. So when checkstyle is requested, gradle\_modules\_worker sets UNSUPPORTED\_TEST to true and returns out of the routine. # Required Functions -* pluginname\_buildfile +## pluginname\_buildfile - - This should be an echo of the file that controls the build system. This is used for module determination. If the build system wishes to disable module determination, it should echo with no args. +This should be an echo of the file that controls the build system. This is used for module determination. If the build system wishes to disable module determination, it should echo with no args. -* pluginname\_executor +## pluginname\_executor - - This should be an echo of how to run the build tool, any extra arguments, etc. +This should be an echo of how to run the build tool, any extra arguments, etc. -* pluginname\_modules\_worker +## pluginname\_modules\_worker - - Input is the branch and the test being run. This should call `modules_workers` with the generic parts to run that test on the build system. For example, if it is convention to use 'test' to trigger 'unit' tests, then `modules_workers` should be called with 'test' appended onto its normal parameters. +Input is the branch and the test being run. This should call `modules_workers` with the generic parts to run that test on the build system. For example, if it is convention to use 'test' to trigger 'unit' tests, then `modules_workers` should be called with 'test' appended onto its normal parameters. -* pluginname\_builtin_personality\_modules +## pluginname\_builtin_personality\_modules - - Default method to determine how to enqueue modules for processing. Note that personalities may override this function. Requires two arguments: repo status and test desired. For example, in a maven build, values may be 'branch' and 'mvninstall'. +Default method to determine how to enqueue modules for processing. Note that personalities may override this function. Requires two arguments: repo status and test desired. For example, in a maven build, values may be 'branch' and 'mvninstall'. -* pluginname\_builtin_personality\_file\_tests +## pluginname\_builtin_personality\_file\_tests - - Default method to determine which tests to trigger. Note that personalities may override this function. Requires a single argument: the file in which the tests exist. +Default method to determine which tests to trigger. Note that personalities may override this function. Requires a single argument: the file in which the tests exist. # Optional Functions -* pluginname\_parse\_args +## pluginname\_parse\_args - - executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment. +Executed prior to any other above functions except for pluginname\_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment. -* pluginname\_initialize +## pluginname\_initialize - - After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc. +After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc. -* pluginname\_reorder\_modules +## pluginname\_reorder\_modules - - This functions allows the plugin to (re-)order the modules (e.g. based on the output of the maven dependency plugin). When called CHANGED\_MODULES[@] already contains all changed modules. It must be altered to have an effect. +This functions allows the plugin to (re-)order the modules (e.g. based on the output of the maven dependency plugin). When called CHANGED\_MODULES[@] already contains all changed modules. It must be altered to have an effect. -* pluginname\_(test)\_logfilter +## pluginname\_(test)\_logfilter - - This functions should filter all lines relevant to this test from the logfile. It is called in preparation for the `calcdiffs` function. The test plug-in name should be in the (test) part of the function name. +This functions should filter all lines relevant to this test from the logfile. It is called in preparation for the `calcdiffs` function. The test plug-in name should be in the (test) part of the function name. -* pluginname\_(test)_calcdiffs +## pluginname\_(test)_calcdiffs - - Some build tools (e.g., maven) use custom output for certain types of compilations (e.g., java). This allows for custom log file difference calculation used to determine the before and after views. +Some build tools (e.g., maven) use custom output for certain types of compilations (e.g., java). This allows for custom log file difference calculation used to determine the before and after views. -* pluginname\_docker\_support +## pluginname\_docker\_support - - If this build tool requires extra settings on the `docker run` command line, this function should be defined and add those options into an array called `${DOCKER_EXTRAARGS[@]}`. This is particularly useful for things like mounting volumes for repository caches. + If this build tool requires extra settings on the `docker run` command line, this function should be defined and add those options into an array called `${DOCKER_EXTRAARGS[@]}`. This is particularly useful for things like mounting volumes for repository caches. - **WARNING**: Be aware that directories that do not exist MAY be created by root by Docker itself under certain conditions. It is HIGHLY recommend that `pluginname_initialize` be used to create the necessary directories prior to be used in the `docker run` command. + **WARNING**: Be aware that directories that do not exist MAY be created by root by Docker itself under certain conditions. It is HIGHLY recommend that `pluginname_initialize` be used to create the necessary directories prior to be used in the `docker run` command. -# Ant Specific +# Apache Ant Specific -## Command Arguments +## Ant Command Arguments test-patch always passes -noinput to Ant. This forces ant to be non-interactive. -## Docker Mode +## Ant in Docker Mode In Docker mode, the `${HOME}/.ivy2` directory is shared amongst all invocations. @@ -140,7 +142,7 @@ In Docker mode, the `${HOME}/.ivy2` directory is shared amongst all invocations. autoconf requires make to be enabled. autoreconf is always used to rebuild the configure scripte. -## Command Arguments +## autoconf Command Arguments autoconf will always run configure with prefix set to a directory in the patch processing directory. To configure other flags, set the AUTCONF_CONF_FLAGS environment variable. @@ -158,13 +160,13 @@ In Docker mode, the `${HOME}/.gradle` directory is shared amongst all invocation No notes. -# Maven Specific +# Apache Maven Specific ## Command Arguments test-patch always passes --batch-mode to maven to force it into non-interactive mode. Additionally, some tests will also force -fae in order to get all of messages/errors during that mode. Some tests are executed with -DskipTests. Additional arguments should be handled via the personality. -## Per-instance Repositories +## Per-instance Repositories Under many common configurations, maven (as of 3.3.3 and lower) may not properly handle being executed by multiple processes simultaneously, especially given that some tests require the `mvn install` command to be used. @@ -174,7 +176,7 @@ By default, `test-patch` uses `${HOME}/yetus-m2` as the base directory to store The location of the `settings.xml` may be changed via the `--mvn-settings` option. -## Docker Mode +## Maven in Docker Mode In Docker mode, `${HOME}/.m2` is shared amongst all invocations. If `--mvn-custom-repos` is used, all of `--mvn-custom-repos-dir` is shared with all invocations. The per-instance directory will be calculated and configured after Docker has launched. diff --git a/asf-site-src/source/documentation/in-progress/precommit-docker-cleanup.md b/asf-site-src/source/documentation/in-progress/precommit-docker-cleanup.md index 9df7524..9add204 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-docker-cleanup.md +++ b/asf-site-src/source/documentation/in-progress/precommit-docker-cleanup.md @@ -17,8 +17,12 @@ under the License. --> -docker-cleanup -============== +# docker-cleanup + + +* [Usage](#usage) + + `docker-cleanup` is a command to perform precommit's Docker cleanup functionality outside of patch and build testing. @@ -40,7 +44,6 @@ Deleted: sha256:1a96c79a0a9ab538c6c7765dc908eca3e689270b778d6ae2add558e89792a7d8 ``` - `docker-cleanup` also supports the `--sentinel` mode to kill and remove stale running containers: ```bash diff --git a/asf-site-src/source/documentation/in-progress/precommit-docker.md b/asf-site-src/source/documentation/in-progress/precommit-docker.md index 8bd344c..1930710 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-docker.md +++ b/asf-site-src/source/documentation/in-progress/precommit-docker.md @@ -17,8 +17,7 @@ under the License. --> -test-patch Docker Support -========================= +# test-patch Docker Support diff --git a/asf-site-src/source/documentation/in-progress/precommit-patchnames.md b/asf-site-src/source/documentation/in-progress/precommit-patchnames.md index ce89b92..51b941c 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-patchnames.md +++ b/asf-site-src/source/documentation/in-progress/precommit-patchnames.md @@ -17,29 +17,35 @@ under the License. --> -We use Apache Yetus to process your patch. It supports the following patterns and -procedures for patch file names: +# Patch File Naming + + + +* [JIRA](#jira) +* [Github](#github) + + + +We use Apache Yetus to process your patch. It supports the following patterns and procedures for patch file names: ## JIRA If JIRA support is configured, attach the patch to the given ISSUE and click 'Submit Patch'. The patch file should be named one of: - * ISSUE.patch - * ISSUE.###.patch - * ISSUE.branch.###.patch - * ISSUE-branch-###.patch - * ISSUE.###.branch.patch - * ISSUE-branch.###.patch +* ISSUE.patch +* ISSUE.###.patch +* ISSUE.branch.###.patch +* ISSUE-branch-###.patch +* ISSUE.###.branch.patch +* ISSUE-branch.###.patch The meaning of the subpart are the following: - * ISSUE: JIRA key (e.g. YETUS-1) - * branch: either the name of a branch or a git hash prefixed with **git** (e.g. branch-2.8 or git8e55427b35) - * ###: revision of the patch (e.g. 00 or 01) +* ISSUE: JIRA key (e.g. YETUS-1) +* branch: either the name of a branch or a git hash prefixed with **git** (e.g. branch-2.8 or git8e55427b35) +* ###: revision of the patch (e.g. 00 or 01) ## Github -If Github support is also configured, a comment that contains a link to a Pull Request's -patch file (e.g., https://github.com/user/repo/pull/1.patch) will pull the patch from -the given Github PR. +If Github support is also configured, a comment that contains a link to a Pull Request's patch file (e.g., ) will pull the patch from the given Github PR. diff --git a/asf-site-src/source/documentation/in-progress/precommit-qbt.md b/asf-site-src/source/documentation/in-progress/precommit-qbt.md index 91d1749..665bd8c 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-qbt.md +++ b/asf-site-src/source/documentation/in-progress/precommit-qbt.md @@ -17,13 +17,19 @@ under the License. --> -qbt -=== +# qbt -`qbt` is a command to execute test-patch without a patch. It uses -the same plug-ins and the same options as test-patch. The only + + +* [Reporting](#reporting) +* [Archiving](#archiving) + + + +`qbt` is a command to execute `test-patch` without a patch. It uses +the same plug-ins and the same options as `test-patch`. The only difference is that no patch file, location, etc should be supplied. -It is meant to be a way to easily get test-patch's output on your +It is meant to be a way to easily get `test-patch`'s output on your current source tree. It is suitable to be run as a regularly scheduled build as part of your overall development strategy. @@ -34,7 +40,7 @@ When using an automation tool, it may be useful to use the file. This can then be used with systems like Jenkin's email-ext plug-in to send the output as an emailed report: -``` +```text ${FILE,path=""} ``` @@ -42,7 +48,7 @@ For something a bit more structured, there is also the `--html-report-file` option. Using this output, again with Jenkins' email-ext plug-in, it is possible to build some very nice looking output that is easily customized: -``` +```html