maven-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Curtis Rueden <ctrue...@wisc.edu>
Subject Re: Best strategy for using lots of modules/projects with some private and some OSS
Date Tue, 22 Sep 2015 19:43:58 GMT
Hi Kevin,

My projects opt for independent versioning of modules to facilitate
"release early, release often." To do this for large sets of components
like yours requires a Bill of Materials -- i.e., common parent POM with
dependencyManagement section.

FWIW, the docs we have about our projects that work this way are at:
* http://imagej.net/Architecture

And in particular:
* http://imagej.net/Architecture#Bill_of_Materials
* http://imagej.net/Architecture#How_SciJava_achieves_reproducible_builds
* http://imagej.net/Philosophy#Release_early.2C_release_often

And the BOM stuff is at:
*
https://github.com/scijava/pom-scijava/blob/pom-scijava-8.3.0/pom.xml#L103-L819

The downside, as you point out, of all components being release version
coupled is that it is annoying to have to do a "release cascade" to
propagate a bug fix from the lowest level components to the highest level
ones. We have some tooling to make that easier (I personally live in the
"releases should be as easy/automated/fast as possible" camp), but the
modularity does cost time sometimes. Hopefully a lot less time than
building your huge multi-module project from scratch every time, though!

I also recently wrote a "melting pot" script to do end-to-end testing of
large component collections:

https://github.com/scijava/scijava-scripts/blob/d892adc0092c220ee1e597b9fb5a1fb067e4509b/melting-pot.sh

This script builds and runs unit tests for all components of a large
collection at their respective versions, all in the same Java runtime, to
ensure that everything _really does_ work together at the versions you are
currently deploying to end users.

I would be happy to know about other tooling people have created to help
with this sort of project structure.

Regards,
Curtis

On Tue, Sep 22, 2015 at 12:47 PM, Kevin Burton <burton@spinn3r.com> wrote:

> We have a multi-module setup whereby we have about 150 independent modules.
>
> Our build takes a long time and actually slows down development as we have
> to do a compile of a LOT of source code to rebuild the project.
>
> Additionally, we have a lot of code that we want to Open Source.
>
> This has meant git submodules the IMO git submodules really don’t work when
> using branches.  They break and require a whole bunch of custom works and
> hack and when they DO break it’s confusing how to resolve them.
>
> This has meant that we’ve not really done a good job of OSSing our code
> base as its just too hard.
>
> What we’ve done to date is just have one major version number across all
> our projects.  So upgrading them and fixing their dependencies means that I
> just have to change a version number everywhere and I’m done.
>
> What I was thinking of is changing this strategy to use the maven
> "versions:use-latest-versions” plugin.
>
> What i would do is have a parent directory named ‘spinn3r’ which just has a
> bunch of git submodules.  We NEVER branch in this directory.
>
> It also means that any of our developers can check it out so that they have
> all of our source code.
>
> At this point I can use a normal development strategy for each project.
> They don’t use submodules which enables us to branch/merge easily.
>
> I can also have a dedicated IntelliJ or Eclipse project for each one and
> switch between them.
>
> Now the main issue I have is how do I bump releases easily and make sure
> all my code is using the latest version of its sibling projects.
>
> In the parent directory I can just run versions:use-latest-versions … on
> each one of the projects so that it automatically pulled in the latest
> version after a release.
>
> The only problem here is that there’s a dependency graph that needs to be
> considered.
>
> for example, if project A depends on project B, then we have to bump the
> version and push project B into maven before we upgrade dependencies on
> project A.
>
> This is a frustrating issue…
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message