hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a.@apache.org
Subject [10/10] hadoop git commit: HADOOP-12130. document features added in 12113 (aw)
Date Fri, 31 Jul 2015 21:53:50 GMT
HADOOP-12130. document features added in 12113 (aw)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/21e21b99
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/21e21b99
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/21e21b99

Branch: refs/heads/HADOOP-12111
Commit: 21e21b990902c99c5bb33464b9b096c28c30edf4
Parents: 659deae
Author: Allen Wittenauer <aw@apache.org>
Authored: Fri Jul 31 14:53:29 2015 -0700
Committer: Allen Wittenauer <aw@apache.org>
Committed: Fri Jul 31 14:53:29 2015 -0700

----------------------------------------------------------------------
 dev-support/docs/precommit-advanced.md     | 68 ++++++++++++++----
 dev-support/docs/precommit-architecture.md | 31 ++++----
 dev-support/docs/precommit-basic.md        | 94 +++++++++++++++++++++----
 3 files changed, 149 insertions(+), 44 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/21e21b99/dev-support/docs/precommit-advanced.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-advanced.md b/dev-support/docs/precommit-advanced.md
index 0a7eac5..a424199 100644
--- a/dev-support/docs/precommit-advanced.md
+++ b/dev-support/docs/precommit-advanced.md
@@ -16,22 +16,35 @@ test-patch
 ==========
 
 * [Docker Support](#Docker_Support)
-* [Maven Profiles](#Maven_Profiles)
+* [Maven Specific](#Maven_Specific)
+* [Ant Specific](#Ant_Specific)
 * [Plug-ins](#Plug-ins)
 * [Configuring for Other Projects](#Configuring_for_Other_Projects)
+* [Important Variables](#Important_Variables)
 
 # Docker Support
 
-By default, test-patch runs in the same shell where it was launched.  It can alternatively
use Docker to launch itself into a container.  This is particularly useful if running under
a QA environment that does not provide all the necessary binaries. For example, the patch
requires a newer version of Java.
+By default, test-patch runs in the same shell where it was launched.  It can alternatively
use Docker to launch itself into a container.  This is particularly useful if running under
a QA environment that does not provide all the necessary binaries. For example, if the patch
requires a newer version of Java.
 
-The `--docker` parameter tells test-patch to run in Docker mode. The `--dockerfile` parameter
allows one to provide a custom Dockerfile.  Be aware that test-patch will copy this file and
append its necessary hooks in order to execute.
+The `--docker` parameter tells test-patch to run in Docker mode. The `--dockerfile` parameter
allows one to provide a custom Dockerfile. The Dockerfile should contain all of the necessary
binaries and tooling needed to run the test.  However be aware that test-patch will copy this
file and append its necessary hooks to re-launch itself prior to executing docker.
 
-test-patch includes code to automatically manage broken/stale container images that are hanging
around if it is run in --jenkins mode.  In this way, if Docker fails to build the image, the
disk space should eventually return.
+Dockerfile images will be named with a test-patch prefix and suffix with either a date or
a git commit hash. By using this information, test-patch will automatically manage broken/stale
container images that are hanging around if it is run in --jenkins mode.  In this way, if
Docker fails to build the image, the disk space should eventually be cleaned and returned
back to the system.
 
-# Maven Profiles
+# Maven Specific
 
-By default, test-patch will pass -Ptest-patch and -D${PROJECT_NAME}PatchProcess to Maven.
This will allow you to configure special actions that should only happen when running underneath
test-patch.
+## Command Arguments
 
+test-patch always passes --batch-mode to maven to force it into non-interactive mode.  Additionally,
some tests will also force -fae in order to get all of messages/errors during that mode. 
It *does not* pass -DskipTests.  Additional arguments should be handled via the personality.
+
+## Test Profile
+
+By default, test-patch will pass -Ptest-patch to Maven. This will allow you to configure
special actions that should only happen when running underneath test-patch.
+
+# Ant Specific
+
+## Command Arguments
+
+test-patch always passes -noinput to Ant.  This force ant to be non-interactive.
 
 # Plug-ins
 
@@ -58,7 +71,7 @@ Similarly, there are other functions that may be defined during the test-patch
r
     - executed prior to the patch being applied but after the git repository is setup.  This
is useful for any early error checking that might need to be done before any heavier work.
 
 * pluginname_preapply
-    - executed prior to the patch being applied.  This is useful for any "before"-type data
collection for later comparisons
+    - executed prior to the patch being applied.  This is useful for any "before"-type data
collection for later comparisons.
 
 * pluginname_postapply
     - executed after the patch has been applied.  This is useful for any "after"-type data
collection.
@@ -79,12 +92,12 @@ If the plug-in has some specific options, one can use following functions:
 
     - executed prior to any other above functions except for pluginname_usage. This is useful
for parsing the arguments passed from the user and setting up the execution environment.
 
-    HINT: It is recommend to make the pluginname relatively small, 10 characters at the most.
 Otherwise the ASCII output table may be skewed.
+    HINT: It is recommended to make the pluginname relatively small, 10 characters at the
most.  Otherwise, the ASCII output table may be skewed.
 
 
 # Configuring for Other Projects
 
-It is impossible for any general framework to be predictive about what types of special rules
any given project may have, especially when it comes to ordering and Maven profiles.  In order
to assist non-Hadoop projects, a project `personality` should be added that enacts these custom
rules.
+It is impossible for any general framework to be predictive about what types of special rules
any given project may have, especially when it comes to ordering and Maven profiles.  In order
to direct test-patch to do the correct action, a project `personality` should be added that
enacts these custom rules.
 
 A personality consists of two functions. One that determines which test types to run and
another that allows a project to dictate ordering rules, flags, and profiles on a per-module,
per-test run.
 
@@ -92,7 +105,7 @@ There can be only **one** of each personality function defined.
 
 ## Test Determination
 
-The `personality_file_tests` function determines which tests to turn on based upon the file
name.  It is realtively simple.  For example, to turn on a full suite of tests for Java files:
+The `personality_file_tests` function determines which tests to turn on based upon the file
name.  It is relatively simple.  For example, to turn on a full suite of tests for Java files:
 
 ```bash
 function personality_file_tests
@@ -131,19 +144,19 @@ function personality_modules
 
 It takes exactly two parameters `repostatus` and `testtype`.
 
-The `repostatus` parameter tells the `personality` function exactly what state the repository
is in.  It can only be in one of two states:  `branch` or `patch`.  `branch` means the patch
has not been applied.  The `patch` state is after the patch has been applied.
+The `repostatus` parameter tells the `personality` function exactly what state the source
repository is in.  It can only be in one of two states:  `branch` or `patch`.  `branch` means
the patch has not been applied.  The `patch` state is after the patch has been applied.
 
 The `testtype` state tells the personality exactly which test is about to be executed.
 
 In order to communicate back to test-patch, there are two functions for the personality to
use.
 
-The first is `clear_personality_queue`. This removes the previous test's configuration so
that a new module queue may be built.
+The first is `clear_personality_queue`. This removes the previous test's configuration so
that a new module queue may be built. Custom personality_modules will almost always want to
do this as the first action.
 
 The second is `personality_enqueue_module`.  This function takes two parameters.  The first
parameter is the name of the module to add to this test's queue.  The second parameter is
an option list of additional flags to pass to Maven when processing it. `personality_enqueue_module`
may be called as many times as necessary for your project.
 
     NOTE: A module name of . signifies the root of the repository.
 
-For example, let's say your project uses a special configuration to skip unit tests (-DskipTests).
 Running unit tests during a javadoc build isn't very interesting. We can write a simple personality
check to disable the unit tests:
+For example, let's say your project uses a special configuration to skip unit tests (-DskipTests).
 Running unit tests during a javadoc build isn't very useful and wastes a lot of time. We
can write a simple personality check to disable the unit tests:
 
 
 ```bash
@@ -160,5 +173,32 @@ function personality_modules
 
 ```
 
-This function will tell test-patch that when the javadoc test is being run, do the documentation
test at the base of the repository and make sure the -DskipTests flag is passed to Maven.
+This function will tell test-patch that when the javadoc test is being run, do the documentation
build at the base of the source repository and make sure the -DskipTests flag is passed to
our build tool.
+
+
+
+# Important Variables
+
+There are a handful of extremely important variables that make life easier for personality
and plug-in writers:
+
+* BUILD\_NATIVE will be set to true if the system has requested that non-JVM-based code be
built (e.g., JNI or other compiled C code). Under Jenkins, this is always true.
+
+* BUILDTOOL specifies which tool is currently being used to drive compilation.  Additionally,
many build tools define xyz\_ARGS to pass on to the build tool command line. (e.g., MAVEN\_ARGS
if maven is in use).  Projects may set this in their personality.  NOTE: today, only one build
tool at a time is supported.  This may change in the future.
+
+* CHANGED\_FILES is a list of all files that appear to be added, deleted, or modified in
the patch.
+
+* CHANGED\_UNFILTERED\_MODULES is a list of all modules that house all of the CHANGED\_FILES.
 Be aware that the root of the source tree is reported as '.'.
+
+* CHANGED\_MODULES reports which modules that appear to have source code in them.
+
+* HOW\_TO\_CONTRIBUTE should be a URL that points to a project's on-boarding documentation
for new users. Currently, it is used to suggest a review of patch naming guidelines. Since
this should be project specific information, it is useful to set in a project's personality.
+
+* ISSUE\_RE is to help test-patch when talking to JIRA.  It helps determine if the given
project is appropriate for the given JIRA issue.
+
+* MODULE and other MODULE\_\* are arrays that contain which modules, the status, etc, to
be operated upon. These should be treated as read-only by plug-ins.
+
+* PATCH\_BRANCH\_DEFAULT is the name of the branch in the git repo that is considered the
master.  This is useful to set in personalities.
+
+* PATCH\_DIR is the name of the temporary directory that houses test-patch artifacts (such
as logs and the patch file itself)
 
+* TEST\_PARALLEL if parallel unit tests have been requested. Project personalities are responsible
for actually enabling or ignoring the request. TEST\_THREADS is the number of threads that
have been requested to run in parallel.

http://git-wip-us.apache.org/repos/asf/hadoop/blob/21e21b99/dev-support/docs/precommit-architecture.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-architecture.md b/dev-support/docs/precommit-architecture.md
index c134728..cd527ae 100644
--- a/dev-support/docs/precommit-architecture.md
+++ b/dev-support/docs/precommit-architecture.md
@@ -14,17 +14,17 @@
 
 # Some Philosophy
 
-* Everyone's time is valuable.  The quicker contributors can get feedback and iterate, the
more likely their contribution will get checked in.  A committer should be able to focus on
the core issues of a contribution rather than details that might be able to be determined
automatically.
+* Everyone's time is valuable.  The quicker contributors can get feedback and iterate, the
more likely and faster their contribution will get checked in.  A committer should be able
to focus on the core issues of a contribution rather than details that can be determined automatically.
 
-* Precommit checks should be fast.  There is no value in testing parts of the source tree
that are not immediately impacted by a change.  Unit testing is the target. They are not a
replacement for full builds, which is where integration tests should happen.
+* Precommit checks should be fast.  There is no value in testing parts of the source tree
that are not immediately impacted by a change.  Unit testing is the target. They are not a
replacement for full builds or integration tests.
 
 * Many open source projects have a desire to have this capability.  Why not generalize a
solution?
 
 * In many build systems (especially with maven), a modular design has been picked.  Why not
leverage that design to make checks faster?
 
-* Projects that use the same language will, with a high degree of certainity, benefit from
the same types of checks.
+* Projects that use the same language will, with a high degree of certainty, benefit from
the same types of checks.
 
-* Portability matters.
+* Portability matters.  Tooling should be as operating system and language agnostic as possible.
 
 # Phases
 
@@ -32,7 +32,7 @@ test-patch works effectively under several different phases:
 
 ## Setup
 
-This is where test-patch configures and validates the environemnt.  Some things done in this
phase:
+This is where test-patch configures and validates the environment.  Some things done in this
phase:
 
 * Defaults
 * Parameter handling
@@ -50,12 +50,12 @@ This acts as a verification of all of the setup parts and is the final
place to
 
 ## Pre-apply
 
-This is where the 'before' work is handled.  Some things done in this phase:
+This is where the 'before' work is handled.  Some things that typically get checked in this
phase:
 
 * The first pass of files and modules that will get patched
-* Validation and information gathering of java, javadoc, site, the mvn repo, findbugs, etc.
+* Validation and information gathering of the source tree pre-patch
 * Author checks
-* check for modified unit tests
+* Check for modified unit tests
 
 ## Patch is Applied
 
@@ -65,15 +65,14 @@ The patch gets applied.  Then a second pass to determine which modules
and files
 
 Now that the patch has been applied, many of the same checks performed in the Pre-apply step
are done again to build an 'after' picture.
 
-* Validation and information gathering of java, javadoc, site, the mvn repo, findbugs, etc.
-
 ## Post-install
 
 Some tests only work correctly when the repo is up-to-date. So
-mvn install is run to update the local repo and we enter this phase.  Tests performed here:
+mvn install is run to update the local repo and we enter this phase.  Some example tests
performed here:
 
-* Verification that maven eclipse integration still works
-* FindBugs
+* javadoc
+* Findbugs
+* Maven eclipse integration still works
 
 ## Unit Tests
 
@@ -81,7 +80,7 @@ Since unit tests are generally the slowest part of the precommit process,
they a
 
 ## Reporting
 
-Finally, the results are reported to the screen and, optionally, to JIRA.
+Finally, the results are reported to the screen and, optionally, to JIRA and/or whatever
bug system has been configured.
 
 # Test Flow
 
@@ -90,8 +89,8 @@ The basic workflow for many of the sub-items in individual phases are:
 1. print a header, so the end user knows that something is happening
 1. verify if the test is needed.  If so, continue on.  Otherwise, return success and let
the next part of the phase execute.
 1. Ask the personality about what modules and what flags should get used
-1. Execute maven in the given modules with the given flags. Log the output and record the
time and result code.
-1. Do any extra work as appropriate (diffs, counts, etc) and either accept the status and
message given by the maven run or change the vote, message, log file, etc.
+1. Execute maven (or some other build tool) in the given modules with the given flags. Log
the output and record the time and result code.
+1. Do any extra work as appropriate (diffs, counts, etc) and either accept the status and
message given by the maven run or change the vote, message, log file, etc, based upon this
extra work.
 1. Add the outcome(s) to the report generator
 
 As one can see, the modules list is one of the key inputs into what actually gets executed.
 As a result, projects must full flexibility in either adding, modifying, or even removing
modules from the test list.  If a personality removes the entire list of modules, then that
test should just be ignored.

http://git-wip-us.apache.org/repos/asf/hadoop/blob/21e21b99/dev-support/docs/precommit-basic.md
----------------------------------------------------------------------
diff --git a/dev-support/docs/precommit-basic.md b/dev-support/docs/precommit-basic.md
index a830cdb..e68ad07 100644
--- a/dev-support/docs/precommit-basic.md
+++ b/dev-support/docs/precommit-basic.md
@@ -18,25 +18,30 @@ test-patch
 * [Purpose](#Purpose)
 * [Pre-requisites](#Pre-requisites)
 * [Basic Usage](#Basic_Usage)
+* [Build Tool](#Build_Tool)
+* [Providing Patch Files](#Providing_Patch_Files)
+* [Project-Specific Capabilities](#Project-Specific_Capabilities)
+* [MultiJDK](#MultiJDK)
+* [Docker](#Docker)
 
-## Purpose
+# Purpose
 
-As part of Hadoop's commit process, all patches to the source base go through a precommit
test that does some (usually) light checking to make sure the proposed change does not break
unit tests and/or passes some other prerequisites.  This is meant as a preliminary check for
committers so that the basic patch is in a known state.  This check, called test-patch, may
also be used by individual developers to verify a patch prior to sending to the Hadoop QA
systems.
+As part of Hadoop's commit process, all patches to the source base go through a precommit
test that does some (relatively) light checking to make sure the proposed change does not
break unit tests and/or passes some other prerequisites such as code formatting guidelines.
 This is meant as a preliminary check for committers so that the basic patch is in a known
state and for contributors to know if they have followed the project's guidelines.  This check,
called test-patch, may also be used by individual developers to verify a patch prior to sending
to the Hadoop QA systems.
 
 Other projects have adopted a similar methodology after seeing great success in the Hadoop
model.  Some have even gone as far as forking Hadoop's precommit code and modifying it to
meet their project's needs.
 
 This is a modification to Hadoop's version of test-patch so that we may bring together all
of these forks under a common code base to help the community as a whole.
 
 
-## Pre-requisites
+# Pre-requisites
 
 test-patch has the following requirements:
 
 * Ant- or Maven-based project (and ant/maven installed)
-* git-based project (and git installed)
+* git-based project (and git 1.7.3 or higher installed)
 * bash v3.2 or higher
 * findbugs 3.x installed
-* shellcheck installed
+* shellcheck installed, preferably 0.3.6 or higher
 * pylint installed
 * GNU diff
 * GNU patch
@@ -57,21 +62,21 @@ Optional:
 * Apache JIRA-based issue tracking
 * JIRA cli tools
 
-The locations of these files are (mostly) assumed to be in the file path, but may be overridden
via command line options.  For Solaris and Solaris-like operating systems, the default location
for the POSIX binaries is in /usr/xpg4/bin.
+The locations of these files are (mostly) assumed to be in the file path, but may be overridden
via command line options.  For Solaris and Solaris-like operating systems, the default location
for the POSIX binaries is in /usr/xpg4/bin and the default location for the GNU binaries is
/usr/gnu/bin.
 
 
-## Basic Usage
+# Basic Usage
 
-This command will execute basic patch testing against a patch file stored in filename:
+This command will execute basic patch testing against a patch file stored in "filename":
 
 ```bash
 $ cd <your repo>
 $ dev-support/test-patch.sh --dirty-workspace --project=projectname <filename>
 ```
 
-The `--dirty-workspace` flag tells test-patch that the repository is not clean and it is
ok to continue.  This version command does not run the unit tests.
+The `--dirty-workspace` flag tells test-patch that the repository is not clean and it is
ok to continue.  By default, unit tests are not run since they may take a significant amount
of time.
 
-To do that, we need to provide the --run-tests command:
+To do turn them on, we need to provide the --run-tests option:
 
 
 ```bash
@@ -85,16 +90,34 @@ A typical configuration is to have two repositories.  One with the code
you are
 
 ```bash
 $ cd <workrepo>
-$ git diff --no-prefix trunk > /tmp/patchfile
+$ git diff master > /tmp/patchfile
 $ cd ../<testrepo>
 $ <workrepo>/dev-support/test-patch.sh --basedir=<testrepo> --resetrepo /tmp/patchfile
 ```
 
 We used two new options here.  --basedir sets the location of the repository to use for testing.
 --resetrepo tells test patch that it can go into **destructive** mode.  Destructive mode
will wipe out any changes made to that repository, so use it with care!
 
-After the tests have run, there is a directory that contains all of the test-patch related
artifacts.  This is generally referred to as the patchprocess directory.  By default, test-patch
tries to make something off of /tmp to contain this content.  Using the `--patchdir` command,
one can specify exactly which directory to use.  This is helpful for automated precommit testing
so that the Jenkins or other automated workflow system knows where to look to gather up the
output.
+After the tests have run, there is a directory that contains all of the test-patch related
artifacts.  This is generally referred to as the patchprocess directory.  By default, test-patch
tries to make something off of /tmp to contain this content.  Using the `--patch-dir` option,
one can specify exactly which directory to use.  This is helpful for automated precommit testing
so that Jenkins or other automated workflow system knows where to look to gather up the output.
 
-## Providing Patch Files
+For example:
+
+```bash
+$ test-patch.sh --jenkins --patch-dir=${WORKSPACE}/patchprocess --basedir=${WORKSPACE}/source
${WORKSPACE}/patchfile
+```
+
+... will trigger test-patch to run in fully automated Jenkins mode, using ${WORKSPACE}/patchprocess
as its scratch space, ${WORKSPACE}/source as the source repository, and ${WORKSPACE}/patchfile
as the name of the patch to test against.
+
+# Build Tool
+
+Out of the box, test-patch is built to use maven.  But what if the project is built using
something else, such as ant?
+
+```bash
+$ test-patch.sh (other options) --build-tool=ant
+```
+
+will tell test-patch to use ant instead of maven to drive the project.
+
+# Providing Patch Files
 
 It is a fairly common practice within the Apache community to use Apache's JIRA instance
to store potential patches.  As a result, test-patch supports providing just a JIRA issue
number.  test-patch will find the *last* attachment, download it, then process it.
 
@@ -106,7 +129,6 @@ $ test-patch.sh (other options) HADOOP-9905
 
 ... will process the patch file associated with this JIRA issue.
 
-
 A new practice is to use a service such as GitHub and its Pull Request (PR) feature.  Luckily,
test-patch supports URLs and many services like GitHub provide ways to provide unified diffs
via URLs.
 
 For example:
@@ -117,6 +139,50 @@ $ test-patch.sh (other options) https://github.com/apache/flink/pull/773.patch
 
 ... will grab a unified diff of PR #773 and process it.
 
+# Project-specific Capabilities
+
+Due to the extensible nature of the system, test-patch allows for projects to define project-specific
rules which we call personalities.  (How to build those rules is covered elsewhere.) There
are two ways to specify which personality to use:
+
+## Direct Method
+
+```bash
+$ test-patch.sh (other options) --personality=(filename)
+```
+
+This tells test-patch to use the personality in the given file.
+
+## Project Method
+
+However, test-patch can detect if it is a personality that is in its "personality" directory
based upon the project name:
+
+```bash
+$ test-patch.sh (other options) --project=(project)
+```
+
+# MultiJDK
+
+For many projects, it is useful to test Java code against multiple versions of JDKs at the
same time.  test-patch can do this with the --multijdkdirs option:
+
+```bash
+$ test-patch.sh (other options) --multijdkdirs="/j/d/k/1,/j/d/k/2"
+```
+
+Not all Java tests support this mode, but those that do will now run their tests with all
of the given versions of Java consecutively (e.g., javac--the Java compliation test).  Tests
that do not support MultiJDK mode (e.g., checkstyle, mvn install) will use JAVA\_HOME.
+
+NOTE: JAVA\_HOME is always appended to the list of JDKs in MultiJDK mode.  If JAVA\_HOME
is in the list, it will be moved to the end.
+
+# Docker
+
+test-patch also has a mode to utilize Docker:
+
+```bash
+$ test-patch.sh (other options) --docker
+```
+
+This will do some preliminary setup and then re-execute itself inside a Docker container.
 For more information on how to provide a custom Dockerfile, see the advanced guide.
+
+
+
 ## In Closing
 
 test-patch has many other features and command line options for the basic user.  Many of
these are self-explanatory.  To see the list of options, run test-patch.sh without any options
or with --help.


Mime
View raw message