hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14453) Split the maven modules into several profiles
Date Thu, 25 May 2017 04:41:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024180#comment-16024180

Allen Wittenauer commented on HADOOP-14453:

bq. Why once-per-day?

Because that's all I need. My general pattern is:

1. Do a pull and merge in changes from trunk overnight
2. Do the mvn install to get local changes down into the appropriate repo while in another
window I look over what actually changed.
3. Review, write a patch, or do whatever, usually in a fresh rebase from trunk into my test
4. Afterward, switch back to trunk and re-build just the jars that were touched by that patch
5. go to step 3

I almost always work exclusively on trunk. On the rare occasions I work in other branches
that are cut off of trunk (can't remember when I last touched branch-2), I use a different
maven repo via MAVEN\_OPTS so there are no jar collisions.

Probably worth noting that dev-support/bin/test-patch and dev-support/bin/qbt with the appropriate
flags automates a lot of this.

bq.  I have 5 git clones in my machine.

Which begs the question: why don't people have 5 maven repos to match?  

maven's shared repo model doesn't really work well with same project/multiple repo dev patterns
because there is nothing to prevent race conditions on it.  Ran two 'mvn installs'?  Woops.
 I wonder which jar those unit tests are running...

A lot of work went into Yetus to specifically work around that issue.  Thus why we haven't
seen any spurious "unknown class" or related issues in testing in over a year now.  They were
directly and completely related to having one maven repo for multiple, simultaneous patch

The community can do what they want with this JIRA (assuming it doesn't break anything). 
I'm just pointing out an alternative that has almost always worked for me, regardless of the

> Split the maven modules into several profiles
> ---------------------------------------------
>                 Key: HADOOP-14453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14453
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: c14453_20170524.patch
> Current all the modules are defined at directly under <project>.  As a result,
we cannot select to build only some of the modules.  We have to build all the modules in any
cases and, unfortunately, it takes a long time.
> We propose split all the modules into multiple profiles so that we could build some of
the modules by disabling some of the profiles.  All the profiles are enabled by default so
that all the modules will be built by default. 
> For example, when we are making change in common.  We could build and run tests under
common by disabling hdfs, yarn, mapreduce, etc. modules.  This will reduce the development
time spend on compiling unrelated modules.
> Note that this is for local maven builds.   We are not proposing to change Jenkins builds,
which always build all the modules.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message