hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-11929) add test-patch plugin points for customizing build layout
Date Mon, 01 Jun 2015 20:00:18 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-11929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Allen Wittenauer updated HADOOP-11929:
--------------------------------------
    Attachment: HADOOP-11929.04.patch

-04:
* big rebase
* includes the current version of HADOOP-11984, just to give the patch some weight

hadoop.sh changes:
* hadoop personality forces tools to be last
* checkstyle should be working now
* mvn install run after javac run
* removed more stuff out of the linux build to match jenkins capabilities.  (we'll build this
back up later)
* removed bzip out of the mac build due to bzip detection in our cmake files being broken
* javadoc pre-reqs now specifically called out
* mvn install adds -DskipTests
* parallel tests support

shellcheck.sh:
* fixes for moved modules

* checkstyle
* significant update to use new per-module framework bits

test-patch.sh:
* fixes for moved modules. CHANGED_MODULES now recalculated after patch application
* New CHANGED_UNFILTERED_MODULES.  CHANGED_MODULES has always had non-code modules stripped
out of it.  CHANGED_UNFILTERED has every module in it.
* new clock_display functions pulls code out of add_jira_table so other routines can print
elapsed time themselves
* jira_footer now has protoc version
* maven log files now print the maven command used at the top of the file
* mvninstall is now separate test type.  (almost, still some todo here)
* reordered test lists in determine_tests
* lots of bug fixes and API changes to the new mvn_module* routines, primarily so mvn_module_messages
can be use for both success and fail, greatly reducing code elsewhere
* much of the time handling has been cleaned up, with reported times now mostly being mvn
times and not test-patch.sh overhead time.  global time reports maven + shell overhead.
* even more file name consistency
* almost all tests now report for *every* module *every* time rather than giving a blanket
+1 for entire chunks


> add test-patch plugin points for customizing build layout
> ---------------------------------------------------------
>
>                 Key: HADOOP-11929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11929
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sean Busbey
>            Assignee: Allen Wittenauer
>            Priority: Minor
>         Attachments: HADOOP-11929.00.patch, HADOOP-11929.01.patch, HADOOP-11929.02.patch,
HADOOP-11929.03.patch, HADOOP-11929.04.patch, hadoop.sh
>
>
> Sean Busbey and I had a chat about this at the Bug Bash. Here's the proposal:
>   * Introduce the concept of a 'personality module'.
>   * There can be only one personality.
>   * Personalities provide a single function that takes as input the name of the test
current being processed
>   * This function uses two other built-in functions to define two queues: maven module
name and profiles to use against those maven module names
>   * If something needs to be compiled prior to this test (but not actually tested), the
personality will be responsible for doing that compilation
> In hadoop, the classic example is hadoop-hdfs needs common compiled with the native bits.
So prior to the javac tests, the personality would check CHANGED_MODULES, see hadoop-hdfs,
and compile common w/ -Pnative prior to letting test-patch.sh do the work in hadoop-hdfs.
Another example is our lack of test coverage of various native bits. Since these require profiles
to be defined prior to compilation, the personality could see that something touches native
code, set the appropriate profile, and let test-patch.sh be on its way.
> One way to think of it is some higher order logic on top of the automated 'figure out
what modules and what tests to run' functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message