hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11929) add test-patch plugin points for customizing build layout
Date Fri, 29 May 2015 18:28:17 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565180#comment-14565180

Chris Nauroth commented on HADOOP-11929:

bq. We could maybe avoid building fuse-dfs, the native mapreduce stuff in trunk, libhadooppipes,
and libwebhdfs unless a file in there had changed. Those subprojects are truly self-contained
so that would work.

I disagree with this statement, at least for the codebase as it stands now.  HDFS-8346 demonstrates
that libwebhdfs has a dependency on libhdfs source files.  There also had been an incorrect
assumption about being able to call libhadoop code.  fuse-dfs links to native_mini_dfs, so
there is a chance that a change in libhdfs could break it.  I'm not sure about libhadooppipes
and the native MR stuff.  I haven't looked at them in a while.

A counter-proposal could be to break up some of these unusual dependencies, or more formally
define a "hadoop-native-common" for the sub-projects to statically link to.  For now though,
I filed HADOOP-11937 to request full native builds so that we catch regressions like HDFS-8346
during pre-commit.

> add test-patch plugin points for customizing build layout
> ---------------------------------------------------------
>                 Key: HADOOP-11929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11929
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sean Busbey
>            Assignee: Allen Wittenauer
>            Priority: Minor
>         Attachments: HADOOP-11929.00.patch, HADOOP-11929.01.patch, HADOOP-11929.02.patch,
HADOOP-11929.03.patch, hadoop.sh
> Sean Busbey and I had a chat about this at the Bug Bash. Here's the proposal:
>   * Introduce the concept of a 'personality module'.
>   * There can be only one personality.
>   * Personalities provide a single function that takes as input the name of the test
current being processed
>   * This function uses two other built-in functions to define two queues: maven module
name and profiles to use against those maven module names
>   * If something needs to be compiled prior to this test (but not actually tested), the
personality will be responsible for doing that compilation
> In hadoop, the classic example is hadoop-hdfs needs common compiled with the native bits.
So prior to the javac tests, the personality would check CHANGED_MODULES, see hadoop-hdfs,
and compile common w/ -Pnative prior to letting test-patch.sh do the work in hadoop-hdfs.
Another example is our lack of test coverage of various native bits. Since these require profiles
to be defined prior to compilation, the personality could see that something touches native
code, set the appropriate profile, and let test-patch.sh be on its way.
> One way to think of it is some higher order logic on top of the automated 'figure out
what modules and what tests to run' functions.

This message was sent by Atlassian JIRA

View raw message