maven-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: Best strategy for using lots of modules/projects with some private and some OSS
Date Fri, 25 Sep 2015 03:09:05 GMT
Pretty impressive.  I think another issue I hadn't thought of is the
overhead of having a CI server/project for each project.

If anything, you have to enumerate each project so you can see inside of it
and what's happening.

Having one macro one means you can see everything in one place but of
course that has drawbacks too.

On Tue, Sep 22, 2015 at 3:28 PM, Ben Podgursky <bpodgursky@gmail.com> wrote:

> +1 for aggressive SNAPSHOT use
>
> Our setup.  YMMV:
>
> - We have ~75 independent projects and each of them is versioned and
> deployed independently.  Few million lines of code.
> - We have CI jenkins builds + unit test suites for every project, and we
> deploy SNAPSHOT artifacts as soon as a build succeeds.
> - Our OSS projects follow the same pattern (https://github.com/liveramp/).
> We do releases every so often for external users, but internally we use
> snapshots which build for every new commit.
> - Every internal dependency is on a SNAPSHOT version.
> - All developers have all code checked out in intellij, source linked
> - If you push changes to a library, you update all downstream usages.  No
> exceptions.  Since you have all source in intellij, this usually isn't
> hard.
>
> Our use-case is a bit different in that we don't really have "application
> versions", since everything is back-end data pipeliens.  But when we
> deploy, we build an artifact with locked SNAPSHOT versions which passes
> integration tests and deploy that artifact.  We deploy many projects
> several times a day.
>
> It takes a bit of diligence, but the payoff of not having to manage
> internal versioning and have formal releases is huge.
>
> On Tue, Sep 22, 2015 at 1:14 PM, Ron Wheeler <
> rwheeler@artifact-software.com
> > wrote:
>
> > +1 (probably better and more complete than my description)
> >
> > Has anyone else looked at using an installer like izPack for assembling
> > test setups?
> > It integrates with Maven and will pick up all the right versions of jars
> > and configuration files and build an installer that will drop the whole
> set
> > where ever you want to test.
> > You can build an installer that will install on different OSs so you can
> > take the same test jar and run it on Windows, Linux or a Mac and get the
> > "right" configuration files for the target OS.
> >
> > Uses maven dependency management to get the jars from your repo and uses
> a
> > command language in XML to assemble the rest of your supporting data and
> > configuration files.
> >
> > A bit smarter than Ant about building application run-time structures but
> > that is all it does.
> >
> > Ron
> >
> >
> > On 22/09/2015 3:43 PM, Curtis Rueden wrote:
> >
> >> Hi Kevin,
> >>
> >> My projects opt for independent versioning of modules to facilitate
> >> "release early, release often." To do this for large sets of components
> >> like yours requires a Bill of Materials -- i.e., common parent POM with
> >> dependencyManagement section.
> >>
> >> FWIW, the docs we have about our projects that work this way are at:
> >> * http://imagej.net/Architecture
> >>
> >> And in particular:
> >> * http://imagej.net/Architecture#Bill_of_Materials
> >> *
> http://imagej.net/Architecture#How_SciJava_achieves_reproducible_builds
> >> * http://imagej.net/Philosophy#Release_early.2C_release_often
> >>
> >> And the BOM stuff is at:
> >> *
> >>
> >>
> https://github.com/scijava/pom-scijava/blob/pom-scijava-8.3.0/pom.xml#L103-L819
> >>
> >> The downside, as you point out, of all components being release version
> >> coupled is that it is annoying to have to do a "release cascade" to
> >> propagate a bug fix from the lowest level components to the highest
> level
> >> ones. We have some tooling to make that easier (I personally live in the
> >> "releases should be as easy/automated/fast as possible" camp), but the
> >> modularity does cost time sometimes. Hopefully a lot less time than
> >> building your huge multi-module project from scratch every time, though!
> >>
> >> I also recently wrote a "melting pot" script to do end-to-end testing of
> >> large component collections:
> >>
> >>
> >>
> https://github.com/scijava/scijava-scripts/blob/d892adc0092c220ee1e597b9fb5a1fb067e4509b/melting-pot.sh
> >>
> >> This script builds and runs unit tests for all components of a large
> >> collection at their respective versions, all in the same Java runtime,
> to
> >> ensure that everything _really does_ work together at the versions you
> are
> >> currently deploying to end users.
> >>
> >> I would be happy to know about other tooling people have created to help
> >> with this sort of project structure.
> >>
> >> Regards,
> >> Curtis
> >>
> >> On Tue, Sep 22, 2015 at 12:47 PM, Kevin Burton <burton@spinn3r.com>
> >> wrote:
> >>
> >> We have a multi-module setup whereby we have about 150 independent
> >>> modules.
> >>>
> >>> Our build takes a long time and actually slows down development as we
> >>> have
> >>> to do a compile of a LOT of source code to rebuild the project.
> >>>
> >>> Additionally, we have a lot of code that we want to Open Source.
> >>>
> >>> This has meant git submodules the IMO git submodules really don’t work
> >>> when
> >>> using branches.  They break and require a whole bunch of custom works
> and
> >>> hack and when they DO break it’s confusing how to resolve them.
> >>>
> >>> This has meant that we’ve not really done a good job of OSSing our code
> >>> base as its just too hard.
> >>>
> >>> What we’ve done to date is just have one major version number across
> all
> >>> our projects.  So upgrading them and fixing their dependencies means
> >>> that I
> >>> just have to change a version number everywhere and I’m done.
> >>>
> >>> What I was thinking of is changing this strategy to use the maven
> >>> "versions:use-latest-versions” plugin.
> >>>
> >>> What i would do is have a parent directory named ‘spinn3r’ which just
> >>> has a
> >>> bunch of git submodules.  We NEVER branch in this directory.
> >>>
> >>> It also means that any of our developers can check it out so that they
> >>> have
> >>> all of our source code.
> >>>
> >>> At this point I can use a normal development strategy for each project.
> >>> They don’t use submodules which enables us to branch/merge easily.
> >>>
> >>> I can also have a dedicated IntelliJ or Eclipse project for each one
> and
> >>> switch between them.
> >>>
> >>> Now the main issue I have is how do I bump releases easily and make
> sure
> >>> all my code is using the latest version of its sibling projects.
> >>>
> >>> In the parent directory I can just run versions:use-latest-versions …
> on
> >>> each one of the projects so that it automatically pulled in the latest
> >>> version after a release.
> >>>
> >>> The only problem here is that there’s a dependency graph that needs to
> be
> >>> considered.
> >>>
> >>> for example, if project A depends on project B, then we have to bump
> the
> >>> version and push project B into maven before we upgrade dependencies on
> >>> project A.
> >>>
> >>> This is a frustrating issue…
> >>>
> >>> --
> >>>
> >>> We’re hiring if you know of any awesome Java Devops or Linux Operations
> >>> Engineers!
> >>>
> >>> Founder/CEO Spinn3r.com
> >>> Location: *San Francisco, CA*
> >>> blog: http://burtonator.wordpress.com
> >>> … or check out my Google+ profile
> >>> <https://plus.google.com/102718274791889610666/posts>
> >>>
> >>>
> >
> > --
> > Ron Wheeler
> > President
> > Artifact Software Inc
> > email: rwheeler@artifact-software.com
> > skype: ronaldmwheeler
> > phone: 866-970-2435, ext 102
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>



-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message