hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-11929) add test-patch plugin points for customizing build layout
Date Sun, 24 May 2015 18:06:17 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-11929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Allen Wittenauer updated HADOOP-11929:
--------------------------------------
    Attachment: HADOOP-11929.00.patch

-00:
* Introduce personality modules
* Move all?/most? Hadoop-specific logic to a personality module, which in turn implements
HADOOP-11937
* Switch all maven bits to have a per-module view of the world instead of global
* Rework the table output, log files, and more to be much more consistent across tests.
* Rework how the clock works
* Utilize OSTYPE everywhere instead of using uname -s directly in a few places
* Remove REQUIRE_TEST_LIB_HADOOP var
* Moved around some functions so they related functions were a bit closer together
* Synced big_console_header, start_clock, and verify_needed_tests in many functions so that
timings and user output were more consistent


NOTES:
* This is (not surprising) a big patch.  It'd be great if people could start looking at it
early to spot problems.  It'd also be useful if HADOOP-11933 could get committed sooner rather
than later since these two patches in their current states probably conflict.  It'll be easier
to rebase this on 11933 than the other way around.
* As I suspected, patch test times went *down* due dropping the requirement of building all
of the modules for javac and javadoc.
* That said, this raises the importance of the nightly build.  I don't necessarily see that
as a bad thing.


TODO:
* checkstyle needs to be reworked
* Figure out why cmake has different output depending for pre- and post -patch (which generates
spurious test-patch results)
* much more testing

> add test-patch plugin points for customizing build layout
> ---------------------------------------------------------
>
>                 Key: HADOOP-11929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11929
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sean Busbey
>            Assignee: Allen Wittenauer
>            Priority: Minor
>         Attachments: HADOOP-11929.00.patch, hadoop.sh
>
>
> Sean Busbey and I had a chat about this at the Bug Bash. Here's the proposal:
>   * Introduce the concept of a 'personality module'.
>   * There can be only one personality.
>   * Personalities provide a single function that takes as input the name of the test
current being processed
>   * This function uses two other built-in functions to define two queues: maven module
name and profiles to use against those maven module names
>   * If something needs to be compiled prior to this test (but not actually tested), the
personality will be responsible for doing that compilation
> In hadoop, the classic example is hadoop-hdfs needs common compiled with the native bits.
So prior to the javac tests, the personality would check CHANGED_MODULES, see hadoop-hdfs,
and compile common w/ -Pnative prior to letting test-patch.sh do the work in hadoop-hdfs.
Another example is our lack of test coverage of various native bits. Since these require profiles
to be defined prior to compilation, the personality could see that something touches native
code, set the appropriate profile, and let test-patch.sh be on its way.
> One way to think of it is some higher order logic on top of the automated 'figure out
what modules and what tests to run' functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message