X-Mailer: ASF-Git Admin Mailer Subject: [17/49] yetus git commit: git hash a444ed18a504a2378f0cd1c6223105f0d1a78416 archived-at: Thu, 13 Jul 2017 21:13:19 -0000 http://git-wip-us.apache.org/repos/asf/yetus/blob/6ca1d766/documentation/0.3.0/precommit-advanced/index.html ---------------------------------------------------------------------- diff --git a/documentation/0.3.0/precommit-advanced/index.html b/documentation/0.3.0/precommit-advanced/index.html deleted file mode 100644 index d7aab7c..0000000 --- a/documentation/0.3.0/precommit-advanced/index.html +++ /dev/null @@ -1,398 +0,0 @@ - - - - - - Apache Yetus - - - - - - - - - - - - - - - -

- - -

test-patch

- -

Docker Support
Plug-ins
Personalities
Important Variables

- -

Docker Support

- -

By default, test-patch runs in the same shell where it was launched. It can alternatively use Docker to launch itself in a container. This is particularly useful if running under a QA environment that does not provide all the necessary binaries. For example, if the patch requires a newer version of Java than what is installed on a Jenkins instance.

- -

The --docker parameter tells test-patch to run in Docker mode. The --dockerfile parameter allows one to provide a custom Dockerfile. The Dockerfile should contain all of the necessary binaries and tooling needed to run the test. test-patch will copy this file up until the text YETUS CUT HERE to a different directory and then append its necessary hooks to re-launch itself prior to executing docker.

- -

If a custom Dockerfile cannot be used or the docker executable does not work, test-patch will attempt to recover by switching to its bundled Dockerfile or disabling docker support and running locally. This behavior can be changed with the --dockeronfail option. It takes a list of comma-delimited settings:

- -

fallback - Use the bundled Dockerfile
continue - Turn off docker support
fail - fail the test

- -

The ‘fail’ setting is always the last option that test-patch will use and may be omitted unless it is the only option.

- -

For example, --dockeronfail=continue means if the Dockerfile can’t be found, just turn off Docker support and continue running. --dockeronfail=fallback will switch to the bundled Dockerfile and then fail the build if docker fails to work. --dockeronfail=fail means to just fail the build and do not try any other mechanisms of recovery. The default is ‘fallback,continue,fail’ which will allow test-patch to try to continue executing as much as it possibily can.

- -

Be aware that if the Dockerfile is found and the docker command works, test-patch will always fail the build if the Dockerfile itself fails the build. It will not attempt to continue in the non-Docker mode.

- -

NOTE: If you are using Boot2Docker, you must use directories under /Users (OSX) or C:\Users (Windows) as the base and patchprocess directories (specified by the –basedir and –patch-dir options respectively), because automatically mountable directories are limited to them. See the Docker documentation.

- -

Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in –robot mode. In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system. The docker mode can also be run in a safe mode that prevents deletions via the --dockerdelrep option. Specifying this option will cause test-patch to only report what it would have deleted, but not actually remove anything.

- -

Plug-ins

- -

test-patch allows one to add to its basic feature set via plug-ins. There is a directory called test-patch.d off of the directory where test-patch.sh lives. Inside this directory one may place some bash shell fragments that, if setup with proper functions, will allow for test-patch to call it as necessary. Different plug-ins have specific functions for that particular functionality. In this document, the common functions available to all/most plug-ins are covered. Test plugins are covered below. See other documentation for pertinent information for the other plug-in types.

- -

Common Plug-in Functions

- -

Every plug-in must have one line in order to be recognized, usually an ‘add’ statement. Test plug-ins, for example, have this statement:

add_test_type <pluginname>
-

This function call registers the pluginname so that test-patch knows that it exists. Plug-in names must be unique across all the different plug-in types. Additionally, the ‘all’ plug-in is reserved. The pluginname also acts as the key to the custom functions that you can define. For example:

function pluginname_filefilter
-

defines the filefilter for the pluginname plug-in.

- -

Similarly, there are other functions that may be defined during the test-patch run:

HINT: It is recommended to make the pluginname relatively small, 10 characters at the most.  Otherwise, the ASCII output table may be skewed.
-

pluginname_usage
- -
- executed when the help message is displayed. This is used to display the plug-in specific options for the user.
pluginname_parse_args
- -
- executed prior to any other above functions except for pluginname_usage. This is useful for parsing the arguments passed from the user and setting up the execution environment.
pluginname_initialize
- -
- After argument parsing and prior to any other work, the initialize step allows a plug-in to do any precursor work, set internal defaults, etc.
pluginname_docker_support
- -
- Perform any necessary setup to configure Docker support for the given plugin. Typically this means adding parameters to the docker run command line via adding to the DOCKER_EXTRAARGS array.
pluginname_precheck
- -
- executed prior to the patch being applied but after the git repository is setup. Returning a fail status here will exit test-patch.
pluginname_patchfile
- -
- executed prior to the patch being applied but after the git repository is setup. This step is intended to perform tests on the content of the patch itself.
pluginname_precompile
- -
- executed prior to the compilation part of the lifecycle. This is useful for doing setup work required by the compilation process.
pluginname_postcompile
- -
- This step happens after the compile phase.
pluginname_rebuild
- -
- Any non-unit tests that require the source to be rebuilt in a destructive way should be run here.

- -

Plug-in Importation

- -

Plug-ins are imported from several key directories:

- -

core.d is an internal-to-Yetus directory that first loads the basic Yetus library, followed by the common routines used by all of the precommit shell code. This order is dictated by prefixing the plug-in files with a number. Other files in this directory are loaded in shell collated order.
personality contains bundled personalities for various projects. These will be imported individually based upon either a project name or if specifically identified with the --personality flag.
test-patch.d contains all of the optional, bundled plug-ins. These are imported last and in shell collated order.

- -

If the --skip-system-plugins flag is passed, then only core.d is imported.

- -

Test Plug-ins

- -

Plug-ins geared towards independent tests are registered via:

add_test_type <pluginname>
-

pluginname_filefilter
- -
- executed while determine which files trigger which tests. This function should use add_test pluginname to add the plug-in to the test list.
pluginname_compile
- -
- executed immediately after the actual compilation. This step is intended to be used to verify the results and add extra checking of the compile phase and it’s stdout/stderr.
pluginname_tests
- -
- executed after the unit tests have completed.
pluginname_clean
- -
- executed to allow the plugin to remove all files that have been generate by this plugin.
pluginname_logfilter
- -
- This functions should filter all lines relevant to this test from the logfile. It is called in preparation for the calcdiffs function.
pluginname_calcdiffs
- -
- This allows for custom log file difference calculation used to determine the before and after views. The default is to use the last column of a colon delimited line of output and perform a diff. If the plug-in does not provide enough context, this may result in error skew. For example, if three lines in a row have Missing period. as the error, test-patch will not be able to determine exactly which line caused the error. Plug-ins that have this issue will want to use this or potentially modify the normal tool’s output (e.g., checkstyle) to provide a more accurate way to determine differences.

- -

NOTE: If the plug-in has support for maven, the maven_add_install pluginname should be executed. See more information in Custom Maven Tests in the build tool documentation.

- -

Personalities

- -

Configuring for Other Projects

- -

It is impossible for any general framework to be predictive about what types of special rules any given project may have, especially when it comes to ordering and Maven profiles. In order to direct test-patch to do the correct action, a project personality should be added that enacts these custom rules.

- -

A personality consists of two functions. One that determines which test types to run and another that allows a project to dictate ordering rules, flags, and profiles on a per-module, per-test run.

- -

There can be only one of each personality function defined.

- -

Global Definitions

- -

Globals for personalities should be defined in the personality_globals function. This function is called after the other plug-ins have been imported. This allows one to configure any settings for plug-ins that have been imported safely:

funciton personality_globals
-{
-  PATCH_BRANCH_DEFAULT=master
-  GITHUB_REPO="apache/yetus"
-}
-

Test Determination

- -

The personality_file_tests function determines which tests to turn on based upon the file name. It is relatively simple. For example, to turn on a full suite of tests for Java files:

function personality_file_tests
-{
-  local filename=$1
-
-  if [[ ${filename} =~ \.java$ ]]; then
-    add_test findbugs
-    add_test javac
-    add_test javadoc
-    add_test mvninstall
-    add_test unit
-  fi
-
-}
-

The add_test function is used to activate the standard tests. Additional plug-ins (such as checkstyle), will get queried on their own.

- -

Module & Profile Determination

- -

Once the tests are determined, it is now time to pick which modules should get used. That’s the job of the personality_modules function.

function personality_modules
-{
-
-    clear_personality_queue
-
-...
-
-    personality_enqueue_module <module> <flags>
-
-}
-

It takes exactly two parameters repostatus and testtype.

- -

The repostatus parameter tells the personality function exactly what state the source repository is in. It can only be in one of two states: branch or patch. branch means the patch has not been applied. The patch state is after the patch has been applied.

- -

The testtype state tells the personality exactly which test is about to be executed.

- -

In order to communicate back to test-patch, there are two functions for the personality to use.

- -

The first is clear_personality_queue. This removes the previous test’s configuration so that a new module queue may be built. Custom personality_modules will almost always want to do this as the first action.

- -

The second is personality_enqueue_module. This function takes two parameters. The first parameter is the name of the module to add to this test’s queue. The second parameter is an option list of additional flags to pass to Maven when processing it. personality_enqueue_module may be called as many times as necessary for your project.

NOTE: A module name of . signifies the root of the repository.
-

For example, let’s say your project uses a special configuration to skip unit tests (-DskipTests). Running unit tests during a javadoc build isn’t very useful and wastes a lot of time. We can write a simple personality check to disable the unit tests:

function personality_modules
-{
-    local repostatus=$1
-    local testtype=$2
-
-    if [[ ${testtype} == 'javadoc' ]]; then
-        personality_enqueue_module . -DskipTests
-        return
-    fi
-    ...
-
-

This function will tell test-patch that when the javadoc test is being run, do the documentation build at the base of the source repository and make sure the -DskipTests flag is passed to our build tool.

- -

Enabling Plug-ins

- -

Personalities can set the base list of plug-ins to enable and disable for their project via the personality_plugins function. Just call it with the same pattern as the --plugins command line option:

personality_plugins "all,-checkstyle,-findbugs,-asflicense"
-

This list is used if the user does not provide a list of plug-ins.

- -

Important Variables

- -

There are a handful of extremely important system variables that make life easier for personality and plug-in writers. Other variables may be provided by individual plug-ins. Check their development documentation for more information.

- -

BUILD_NATIVE will be set to true if the system has requested that non-JVM-based code be built (e.g., JNI or other compiled C code). Under Jenkins, this is always true.
BUILDTOOL specifies which tool is currently being used to drive compilation. Additionally, many build tools define xyz_ARGS to pass on to the build tool command line. (e.g., MAVEN_ARGS if maven is in use). Projects may set this in their personality. NOTE: today, only one build tool at a time is supported. This may change in the future.
CHANGED_FILES[@] is an array of all files that appear to be added, deleted, or modified in the patch.
CHANGED_MODULES[@] is an array of all modules that house all of the CHANGED_FILES[@]. Be aware that the root of the source tree is reported as ‘.’.
DOCKER_EXTRAARGS[@] is an array of command line arguments to apply to the docker run command.
GITHUB_REPO is to help test-patch when talking to Github. If test-patch is given just a number on the command line, it will default to using this repo to determine the pull request.
JIRA_ISSUE_RE is to help test-patch when talking to JIRA. It helps determine if the given project is appropriate for the given JIRA issue.
MODULE and other MODULE_* are arrays that contain which modules, the status, etc, to be operated upon. These should be treated as read-only by plug-ins.
PATCH_BRANCH_DEFAULT is the name of the branch in the git repo that is considered the master. This is useful to set in personalities.
PATCH_DIR is the name of the temporary directory that houses test-patch artifacts (such as logs and the patch file itself)
PATCH_NAMING_RULE should be a URL that points to a project’s on-boarding documentation for new users. It is used to suggest a review of patch naming guidelines. Since this should be project specific information, it is useful to set in a project’s personality.
TEST_PARALLEL if parallel unit tests have been requested. Project personalities are responsible for actually enabling or ignoring the request. TEST_THREADS is the number of threads that have been requested to run in parallel.

- -

- - - http://git-wip-us.apache.org/repos/asf/yetus/blob/6ca1d766/documentation/0.3.0/precommit-apidocs/core/index.html ---------------------------------------------------------------------- diff --git a/documentation/0.3.0/precommit-apidocs/core/index.html b/documentation/0.3.0/precommit-apidocs/core/index.html deleted file mode 100644 index 8fed60e..0000000 --- a/documentation/0.3.0/precommit-apidocs/core/index.html +++ /dev/null @@ -1,1292 +0,0 @@ - - - - - - Apache Yetus - - - - - - - - - - - - - - - -

- - -

Public/Stable/Replaceable - -
- add_bugsystem
- add_build_tool
- add_test
- add_test_format
- add_test_type
- delete_bugsystem
- delete_build_tool
- delete_test
- delete_test_format
- delete_test_type
- personality_plugins
- verify_needed_test
- verify_plugin_enabled
Public/Stable/Not Replaceable - -
- common_defaults
- patchfile_verify_zero
- yetus_abs
- yetus_add_entry
- yetus_debug
- yetus_delete_entry
- yetus_error
- yetus_run_and_redirect
- yetus_verify_entry
Public/Evolving/Not Replaceable - -
None/None/Not Replaceable - -
- docker_version
- personality_file_tests
- personality_modules

- -

Public/Stable/Replaceable

- -

`add_bugsystem`

- -

Synopsis

add_bugsystem bugsystem
-

Description

- -

Add the given bugsystem type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`add_build_tool`

- -

Synopsis

add_build_tool build tool
-

Description

- -

Add the given build tool type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`add_test`

- -

Synopsis

add_test test
-

Description

- -

Add the given test type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`add_test_format`

- -

Synopsis

add_test_format test format
-

Description

- -

Add the given test format type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`add_test_type`

- -

Synopsis

add_test_type plugin
-

Description

- -

Add the given test type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`delete_bugsystem`

- -

Synopsis

delete_bugsystem bugsystem
-

Description

- -

Remove the given bugsystem type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`delete_build_tool`

- -

Synopsis

delete_build_tool build tool
-

Description

- -

Remove the given build tool type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`delete_test`

- -

Synopsis

delete_test test
-

Description

- -

Remove the given test type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`delete_test_format`

- -

Synopsis

delete_test_format test format
-

Description

- -

Remove the given test format type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`delete_test_type`

- -

Synopsis

delete_test_type plugin
-

Description

- -

Remove the given test type

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`personality_plugins`

- -

Synopsis

personality_plugins plug-in list string
-

Description

- -

Personality-defined plug-in list

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`verify_needed_test`

- -

Synopsis

verify_needed_test test
-

Description

- -

Verify if a given test was requested

- -

Returns

- -

0 = yes

- -

1 = no

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

`verify_plugin_enabled`

- -

Synopsis

verify_plugin_enabled test
-

Description

- -

Determine if a plugin was enabeld by the user ENABLED_PLUGINS must be defined

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	Yes

Public/Stable/Not Replaceable

- -

`common_defaults`

- -

Synopsis

common_defaults
-

Description

- -

Setup the default global variables

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	No

`patchfile_verify_zero`

- -

Synopsis

patchfile_verify_zero log filename
-

Description

- -

if patch-level zero, then verify we aren’t just adding files

- -

Returns

- -

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	No

`yetus_abs`

- -

Synopsis

yetus_abs fsobj
-

Description

- -

Given a filename or dir, return the absolute version of it

- -

Returns

- -

0 success

- -

1 failure

- -

stdout abspath

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	No

`yetus_add_entry`

- -

Synopsis

yetus_add_entry
-

Description

- -

Given variable $1 add $2 to it

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	No

`yetus_debug`

- -

Synopsis

yetus_debug string
-

Description

- -

Print a message to stderr if –debug is turned on

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	No

`yetus_delete_entry`

- -

Synopsis

yetus_delete_entry
-

Description

- -

Given variable $1 delete $2 from it

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	No

`yetus_error`

- -

Synopsis

yetus_error string
-

Description

- -

Print a message to stderr

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	No

`yetus_run_and_redirect`

- -

Synopsis

yetus_run_and_redirect filename command [..]
-

Description

- -

run the command, sending stdout and stderr to the given filename

- -

Returns

- -

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	No

`yetus_verify_entry`

- -

Synopsis

yetus_verify_entry
-

Description

- -

Given variable $1 determine if $2 is in it

- -

Returns

- -

0 = yes, 1 = no

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Stable
Replaceable	No

Public/Evolving/Not Replaceable

- -

`list_plugins`

- -

Synopsis

list_plugins
-

Description

- -

List all installed plug-ins, regardless of whether they have been enabled

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Evolving
Replaceable	No

`plugin_usage_output`

- -

Synopsis

plugin_usage_output array
-

Description

- -

Print the plugin’s usage info

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Evolving
Replaceable	No

`yetus_add_header`

- -

Synopsis

yetus_add_header header
-

Description

- -

Add a header to the usage output

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Evolving
Replaceable	No

`yetus_add_option`

- -

Synopsis

yetus_add_option subcommand subcommanddesc
-

Description

- -

Add an option to the usage output

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Evolving
Replaceable	No

`yetus_comma_to_array`

- -

Synopsis

yetus_comma_to_array arrayname string
-

Description

- -

Convert a comma-delimited string to an array

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Evolving
Replaceable	No

`yetus_generic_columnprinter`

- -

Synopsis

yetus_generic_columnprinter array
-

Description

- -

Print a screen-size aware two-column output

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	Public
Stability	Evolving
Replaceable	No

None/None/Not Replaceable

- -

`docker_version`

- -

Synopsis

docker_version
-

Description

- -

None

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	None
Stability	None
Replaceable	No

`personality_file_tests`

- -

Synopsis

personality_file_tests
-

Description

- -

None

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	None
Stability	None
Replaceable	No

`personality_modules`

- -

Synopsis

personality_modules
-

Description

- -

None

- -

Returns

- -

Nothing

- - - - - - - - - - - - - - - - -

Classification	Level
Audience	None
Stability	None
Replaceable	No

- -

- - - http://git-wip-us.apache.org/repos/asf/yetus/blob/6ca1d766/documentation/0.3.0/precommit-apidocs/index.html ---------------------------------------------------------------------- diff --git a/documentation/0.3.0/precommit-apidocs/index.html b/documentation/0.3.0/precommit-apidocs/index.html deleted file mode 100644 index 5940a91..0000000 --- a/documentation/0.3.0/precommit-apidocs/index.html +++ /dev/null @@ -1,153 +0,0 @@ - - - - - - Apache Yetus - - - - - - - - - - - - - - - -

- - -

Precommit API Docs

- -

Following gives the downstream consumable API for precommit components.

- -

Core Functions, common to both smart-apply-patch and test-patch.
Smart Apply Patch
Test Patch, available to all plugins.
Precommit Plugins

- -

- - -