yetus-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a.@apache.org
Subject [yetus] branch master updated: YETUS-109. documentation: clarify --plugin
Date Fri, 04 Jan 2019 01:39:05 GMT
This is an automated email from the ASF dual-hosted git repository.

aw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yetus.git


The following commit(s) were added to refs/heads/master by this push:
     new ed57f34  YETUS-109. documentation: clarify --plugin
ed57f34 is described below

commit ed57f3467c0705f5c06657bca360a705752a89e0
Author: Allen Wittenauer <aw@apache.org>
AuthorDate: Thu Dec 27 22:39:03 2018 -0800

    YETUS-109. documentation: clarify --plugin
    
    Signed-off-by: Sean Busbey <busbey@apache.org>
---
 .../documentation/in-progress/precommit-basic.md   | 158 +++++++++++++--------
 1 file changed, 102 insertions(+), 56 deletions(-)

diff --git a/asf-site-src/source/documentation/in-progress/precommit-basic.md b/asf-site-src/source/documentation/in-progress/precommit-basic.md
index 63d25f2..0085bc9 100644
--- a/asf-site-src/source/documentation/in-progress/precommit-basic.md
+++ b/asf-site-src/source/documentation/in-progress/precommit-basic.md
@@ -22,7 +22,9 @@ test-patch
 
 * [Purpose](#purpose)
 * [Pre-requisites](#pre-requisites)
-* [Basic Usage](#basic-usage)
+* [First Steps](#first-steps)
+* [Resetting the Repository](#resetting-the-repository)
+* [Enabling Features](#enabling-features)
 * [Output Directory](#output-directory)
 * [Build Tool](#build-tool)
 * [Providing Patch Files](#providing-patch-files)
@@ -40,7 +42,6 @@ Other projects have adopted a similar methodology after seeing great success
in
 One of the key facets of Apache Yetus is to bring together all of these forks under a common
code base to help software development
 as a whole.
 
-
 # Pre-requisites
 
 `test-patch` and `smart-apply-patch` are written in bash for maximum portability.  As such,
it mostly assumes the locations of commands to be in the file path. However, in many cases,
this assumption may be overridden via command line options.
@@ -101,21 +102,71 @@ Language Support, Licensing, and more:
 * [checkstyle](http://checkstyle.sourceforge.net/) entries in build system (ant and maven
only)
 * [FindBugs](http://findbugs.sourceforge.net/) entries in build system and 3.x executables
    (NOTE: FindBugs executables are required even if the build system is using [Spotbugs](https://spotbugs.github.io/))
+* [jshint](https://jshint.com) installed
+* [hadolint](https://github.com/hadolint/hadolint) installed
 * [Perl::Critic](http://perlcritic.com/) installed
 * [pylint](http://www.pylint.org/) installed
 * [rubocop](http://batsov.com/rubocop/) installed
 * [ruby-lint](https://github.com/YorickPeterse/ruby-lint) installed
 * [shellcheck](https://github.com/koalaman/shellcheck) installed, preferably 0.3.6 or higher
+* [yamllint](https://github.com/adrienverge/yamllint) installed
+
+# First Steps
+
+A typical local configuration is to have two repositories.  One with the code you are working
on and another, clean repository.  The workflow looks similar to the following:
+
+```bash
+$ cd <workrepo>
+$ git format-patch master > /tmp/patchfile
+$ test-patch --basedir=/some/path/testrepo /tmp/patchfile
+```
+
+ 1. In your work repo, generate a patch.
+ 2. Run your patch against the test repository.
+
+`--basedir` sets the location of the repository to use for testing.   The `/tmp/patchfile`
is the patch file that is generated by `git format-patch` or `git diff` or any number of other
types of input.
+
+# Resetting the Repository
+
+After running that command, the test repository will contain the remnants of the build for
later debugging.  It is inconvenient to instead use:
+
+```bash
+$ cd <workrepo>
+$ git format-patch master > /tmp/patchfile
+$ test-patch --basedir=/some/path/testrepo --resetrepo /tmp/patchfile
+```
 
-# Basic Usage
+ `--resetrepo` tells test patch that it can go into **destructive** mode.  Destructive mode
will wipe out any changes made to that repository, so use it with care!
 
-The first step for a successful deployment is determining which features/plug-ins to enable:
+# Enabling Features
+
+In general, almost all features are enabled via the plug-in framework.  To see what has been
enabled, use the `--list-plugins` option:
 
 ```bash
 $ test-patch --list-plugins
 ```
 
-This option will list all of the available plug-ins that are installed in the default location.
 From this list, the specific plug-ins can be enabled:
+You should see output similar to this:
+
+```
+BUILDTOOLS:
+  ant autoconf cmake gradle make maven nobuild
+TESTTYPES:
+  asflicense author cc checkstyle dupname findbugs hadolint javac javadoc jshint mvnsite
pathlen perlcritic pylint rubocop ruby_lint scalac scaladoc shellcheck shelldocs test4tests
unitveto whitespace xml yamllint
+BUGSYSTEMS:
+  briefreport bugzilla github gitlab htmlout jira
+TESTFORMATS:
+  ctest junit tap
+```
+
+This output shows all of the plug-ins that are available as well as what type of plug-in
they are.
+
+* BUILDTOOLS - These drive the actual compiling of the source tree.
+* TESTTYPES - Various kinds of tests that precommit can apply to the source, patches, and
generated output (object files, etc).
+* BUGSYSTEMS - Integration support for input of patches or output of reports.
+* TESTFORMATS - Types of unit test output that precommit reads.
+
+From this list, the specific plug-ins can be enabled:
 
 ```bash
 $ test-patch --plugins="ant,maven,shellcheck,xml" <other options>
@@ -135,13 +186,13 @@ $ test-patch --plugins="all,-checkstyle,-findbugs" <other options>
 
 This will enable all plug-ins for potential usage, except for checkstyle and findbugs.
 
-**NOTE: The examples in this section will assume that the necessary `--plugins` option has
been set on the command line as appropriate for your particular installation.**
+**NOTE: Many examples in this section will use `--plugins=all`. Users should set the `--plugins`
option as appropriate.**
 
 This command will execute basic patch testing against a patch file stored in "filename":
 
 ```bash
 $ cd <your repo>
-$ test-patch --dirty-workspace --project=projectname <filename>
+$ test-patch --plugins=all --dirty-workspace --project=projectname <filename>
 ```
 
 The `--dirty-workspace` flag tells test-patch that the repository is not clean and it is
ok to continue.  By default, unit tests are not run since they may take a significant amount
of time.
@@ -150,42 +201,19 @@ To do turn them on, we need to provide the --run-tests option:
 
 ```bash
 $ cd <your repo>
-$ test-patch --dirty-workspace --run-tests <filename>
+$ test-patch --plugins=all  --dirty-workspace --run-tests <filename>
 ```
 
 This is the same command, but now runs the unit tests.
 
-A typical configuration is to have two repositories.  One with the code you are working on
and another, clean repository.  This means you can:
-
-```bash
-$ cd <workrepo>
-$ git diff master > /tmp/patchfile
-$ cd ../<testrepo>
-$ test-patch --basedir=<testrepo> --resetrepo /tmp/patchfile
-```
-
-We used two new options here.  `--basedir` sets the location of the repository to use for
testing.  `--resetrepo` tells test patch that it can go into **destructive** mode.  Destructive
mode will wipe out any changes made to that repository, so use it with care!
-
-# Fork Bomb Protection
-
-By default, `test-patch` will set the user soft limit (`ulimit -Su`) to a relatively low
1,000 processes (and, on some operating systems with some languages such as Java, threads!).
This is to prevent errant processes from eating up all system resources.  If this limit is
too low, it may be necessary to use the `--proclimit` option.  For example:
-
-```bash
-$ test-patch --proclimit=10000
-```
-
-... will set it to be 10,000 processes.
-
-  NOTE: The actual implementation of this feature is dependent upon the version of Bash.
 For bash v4 and higher (most operating systems), the fork bomb protection is generally only
used for the build and QA tools.  This means Apache Yetus should continue to function. For
earlier versions of bash (e.g., OS X), the limit is applied to all of test-patch. If the limit
is hit, Apache Yetus will itself likely crash.
-
 # Output Directory
 
-After the tests have run, there is a directory that contains all of the `test-patch` related
artifacts.  This is generally referred to as the patch directory.  By default, `test-patch`
tries to make something off of /tmp to contain this content.  Using the `--patch-dir` option,
one can specify exactly which directory to use.  This is helpful for automated precommit testing
so that [continuous integration systems](../precommit-robots) knows where to look to gather
up the output.
+After the tests have run, there is a directory that contains all of the `test-patch` related
artifacts.  This is generally referred to as the patch directory.  By default, `test-patch`
tries to make something off of `/tmp` to contain this content.  Using the `--patch-dir` option,
one can specify exactly which directory to use.  This is helpful for automated precommit testing
so that [continuous integration systems](../precommit-robots) knows where to look to gather
up the output.
 
 For example:
 
 ```bash
-$ test-patch --patch-dir=${WORKSPACE}/patchdir --basedir=${WORKSPACE}/source ${WORKSPACE}/patchfile
+$ test-patch --plugins=all --patch-dir=${WORKSPACE}/patchdir --basedir=${WORKSPACE}/source
${WORKSPACE}/patchfile
 ```
 
 ... will trigger `test-patch` to run in fully automated mode, using `${WORKSPACE}/patchdir`
as its scratch space, `${WORKSPACE}/source` as the source repository, and `${WORKSPACE}/patchfile`
as the name of the patch to test against.  This will always run the unit tests, write answers
back to bug systems, remove old, stopped/exited Docker containers after 24 hours and images
after 1 week, forcibly use `--resetrepo`, and more.
@@ -194,10 +222,10 @@ $ test-patch --patch-dir=${WORKSPACE}/patchdir --basedir=${WORKSPACE}/source
${W
 
 # Build Tool
 
-Out of the box, test-patch will try to figure out which build tool the project uses.  But
what if you want to override it?  The `--build-tool` option allows a manual setting:
+Out of the box, `test-patch` will try to figure out which build tool the project uses.  But
what if you want to override it?  The `--build-tool` option allows a manual setting:
 
 ```bash
-$ test-patch (other options) --build-tool=ant
+$ test-patch --plugins=all --build-tool=ant (other options)
 ```
 
 will tell `test-patch` to use `ant` instead of maven to drive the project.
@@ -205,7 +233,7 @@ will tell `test-patch` to use `ant` instead of maven to drive the project.
 To disable the build tool entirely, use the `nobuild` setting:
 
 ```bash
-$ test-patch (other options) --build-tool=nobuild
+$ test-patch --plugins=all --build-tool=nobuild (other options)
 ```
 
 # Providing Patch Files
@@ -214,7 +242,7 @@ NOTE: More in-depth information may be found in the [bugsystems](../precommit-bu
 
 ## JIRA
 
-It is a fairly common practice within the Apache community to use Apache's JIRA instance
to store potential patches.  As a result, test-patch supports providing just a JIRA issue
number.  test-patch will find the *last* attachment, download it, then process it.
+It is a fairly common practice within the Apache community to use Apache's JIRA instance
to store potential patches.  As a result, `test-patch` supports providing just a JIRA issue
number via the `jira` plug-in.  `test-patch` will find the *last* attachment, download it,
then process it.
 
 **NOTE: `test-patch` expects the patch files to follow a particular naming convention. For
complete details
  on the naming convention please refer to [patch-naming-conventions](../precommit-patchnames/)**
@@ -222,7 +250,7 @@ It is a fairly common practice within the Apache community to use Apache's
JIRA
 For example:
 
 ```bash
-$ test-patch (other options) HADOOP-9905
+$ test-patch --plugins=all HADOOP-9905 (other options)
 ```
 
 ... will process the patch file associated with this JIRA issue.
@@ -230,51 +258,51 @@ $ test-patch (other options) HADOOP-9905
 If the Apache JIRA system is not in use, then override options may be provided on the command
line to point to a different JIRA instance.
 
 ```bash
-$ test-patch --jira-issue-re='^PROJECT-[0-9]+$' --jira-base-url='https://example.com/jira'
PROJECT-90
+$ test-patch --plugins=all --jira-issue-re='^PROJECT-[0-9]+$' --jira-base-url='https://example.com/jira'
PROJECT-90
 ```
 
 ... will process the patch file attached to PROJECT-90 on the JIRA instance located on the
example.com server.
 
 ## GITHUB
 
-`test-patch` has built-in support for Github.  `test-patch` supports many forms of providing
pull requests to work on:
+`test-patch` has built-in support for Github via the `github` plug-in.  `test-patch` supports
many forms of providing pull requests to work on:
 
 ```bash
-$ test-patch --github-repo=apache/pig GH:99
+$ test-patch --plugins=all --github-repo=apache/pig GH:99
 ```
 
 or
 
 ```bash
-$ test-patch https://github.com/apache/pig/pulls/99
+$ test-patch --plugins=all https://github.com/apache/pig/pulls/99
 ```
 
 or
 
 ```bash
-$ test-patch https://github.com/apache/pig/pulls/99.patch
+$ test-patch --plugins=all https://github.com/apache/pig/pulls/99.patch
 ```
 
 ... will process PR #99 on the apache/pig repo.
 
 ## GITLAB
 
-`test-patch` has support for Gitlab.  `test-patch` supports many forms of providing merge
requests to work on:
+`test-patch` has support for Gitlab via the `gitlab` plug-in.  `test-patch` supports many
forms of providing merge requests to work on:
 
 ```bash
-$ test-patch --gitlab-repo=_a__w_/yetus GL:1
+$ test-patch --plugins=all --gitlab-repo=_a__w_/yetus GL:1
 ```
 
 or
 
 ```bash
-$ test-patch https://gitlab.com/_a__w_/yetus/merge_requests/3
+$ test-patch --plugins=all https://gitlab.com/_a__w_/yetus/merge_requests/3
 ```
 
 or
 
 ```bash
-$ test-patch https://gitlab.com/_a__w_/yetus/merge_requests/3.patch
+$ test-patch --plugins=all https://gitlab.com/_a__w_/yetus/merge_requests/3.patch
 ```
 
 ... will process MR #3 on the \_a\_\_w\_/yetus repo.
@@ -282,12 +310,12 @@ $ test-patch https://gitlab.com/_a__w_/yetus/merge_requests/3.patch
 
 ## Generic URLs
 
-Luckily, test-patch supports ways to provide unified diffs via URLs.
+Luckily, `test-patch` supports ways to provide unified diffs via URLs.
 
 For example:
 
 ```bash
-$ test-patch (other options) https://example.com/webserver/file.patch
+$ test-patch --plugins=all https://example.com/webserver/file.patch
 ```
 
 ... will download and process the file.patch from the example.com webserver.
@@ -299,25 +327,38 @@ Due to the extensible nature of the system, `test-patch` allows for
projects to
 ## Direct Method
 
 ```bash
-$ test-patch (other options) --personality=(filename)
+$ test-patch --plugins=all --personality=(filename)
 ```
 
-This tells test-patch to use the personality in the given file.
+This tells `test-patch` to use the personality in the given file.
 
 ## Project Method
 
 However, `test-patch` can detect if it is a personality that is in its "personality" directory
based upon the project name:
 
 ```bash
-$ test-patch (other options) --project=(project)
+$ test-patch --plugins=all --project=(project)
 ```
 
+# Fork Bomb Protection
+
+By default, `test-patch` will set the user soft limit (`ulimit -Su`) to a relatively low
1,000 processes (and, on some operating systems with some languages such as Java, threads!).
This is to prevent errant processes from eating up all system resources.  If this limit is
too low, it may be necessary to use the `--proclimit` option.  For example:
+
+```bash
+$ test-patch --plugins=all --proclimit=10000
+```
+
+... will set it to be 10,000 processes.
+
+  NOTE: The actual implementation of this feature is dependent upon the version of Bash.
 For bash v4 and higher (most operating systems), the fork bomb protection is generally only
used for the build and QA tools.  This means Apache Yetus should continue to function. For
earlier versions of bash (e.g., OS X), the limit is applied to all of test-patch. If the limit
is hit, Apache Yetus will itself likely crash.
+
+
 # MultiJDK
 
-For many projects, it is useful to test Java code against multiple versions of JDKs at the
same time.  test-patch can do this with the --multijdkdirs option:
+For many projects, it is useful to test Java code against multiple versions of JDKs at the
same time.  In combination with the `java` plug-in, `test-patch` can do this with the `--multijdkdirs`
option:
 
 ```bash
-$ test-patch (other options) --multijdkdirs="/j/d/k/1,/j/d/k/2"
+$ test-patch --plugins=all --multijdkdirs="/j/d/k/1,/j/d/k/2"
 ```
 
 Not all Java tests support this mode, but those that do will now run their tests with all
of the given versions of Java consecutively (e.g., `javac`--the Java compliation test).  Tests
that do not support MultiJDK mode (e.g., checkstyle, mvn install) will use JAVA\_HOME.
@@ -326,14 +367,19 @@ NOTE: JAVA\_HOME is always appended to the list of JDKs in MultiJDK
mode.  If JA
 
 # Docker
 
-`test-patch` also has a mode to utilize Docker:
+`test-patch` also has a built-in mode (i.e., no plug-in required) to utilize Docker:
 
 ```bash
+<<<<<<< HEAD
 $ test-patch (other options) --docker
+=======
+$ test-patch --docker
+>>>>>>> YETUS-109. documentation: clarify --plugin
 ```
 
 This command will do some preliminary setup and then re-execute itself inside a Docker container.
 For more information on how to provide a custom Dockerfile and other Docker-specific features,
see the specific [precommit Docker support](../precommit-docker) page and the [Apache Yetus
Docker Hub Images](/yetus-docker-image) page for more information on the convenience Docker
images.
 
+
 # In Closing
 
-test-patch has many other features and command line options for the basic user.  Many of
these are self-explanatory.  To see the list of options, run `test-patch` without any options
or with --help.
+`test-patch` has many other features and command line options for the basic user.  Many of
these are self-explanatory.  To see the list of options, run `test-patch` without any options
or with `--help`.


Mime
View raw message