maven-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Podgursky <bpodgur...@gmail.com>
Subject Re: Best strategy for using lots of modules/projects with some private and some OSS
Date Tue, 22 Sep 2015 22:28:38 GMT
+1 for aggressive SNAPSHOT use

Our setup.  YMMV:

- We have ~75 independent projects and each of them is versioned and
deployed independently.  Few million lines of code.
- We have CI jenkins builds + unit test suites for every project, and we
deploy SNAPSHOT artifacts as soon as a build succeeds.
- Our OSS projects follow the same pattern (https://github.com/liveramp/).
We do releases every so often for external users, but internally we use
snapshots which build for every new commit.
- Every internal dependency is on a SNAPSHOT version.
- All developers have all code checked out in intellij, source linked
- If you push changes to a library, you update all downstream usages.  No
exceptions.  Since you have all source in intellij, this usually isn't hard.

Our use-case is a bit different in that we don't really have "application
versions", since everything is back-end data pipeliens.  But when we
deploy, we build an artifact with locked SNAPSHOT versions which passes
integration tests and deploy that artifact.  We deploy many projects
several times a day.

It takes a bit of diligence, but the payoff of not having to manage
internal versioning and have formal releases is huge.

On Tue, Sep 22, 2015 at 1:14 PM, Ron Wheeler <rwheeler@artifact-software.com
> wrote:

> +1 (probably better and more complete than my description)
>
> Has anyone else looked at using an installer like izPack for assembling
> test setups?
> It integrates with Maven and will pick up all the right versions of jars
> and configuration files and build an installer that will drop the whole set
> where ever you want to test.
> You can build an installer that will install on different OSs so you can
> take the same test jar and run it on Windows, Linux or a Mac and get the
> "right" configuration files for the target OS.
>
> Uses maven dependency management to get the jars from your repo and uses a
> command language in XML to assemble the rest of your supporting data and
> configuration files.
>
> A bit smarter than Ant about building application run-time structures but
> that is all it does.
>
> Ron
>
>
> On 22/09/2015 3:43 PM, Curtis Rueden wrote:
>
>> Hi Kevin,
>>
>> My projects opt for independent versioning of modules to facilitate
>> "release early, release often." To do this for large sets of components
>> like yours requires a Bill of Materials -- i.e., common parent POM with
>> dependencyManagement section.
>>
>> FWIW, the docs we have about our projects that work this way are at:
>> * http://imagej.net/Architecture
>>
>> And in particular:
>> * http://imagej.net/Architecture#Bill_of_Materials
>> * http://imagej.net/Architecture#How_SciJava_achieves_reproducible_builds
>> * http://imagej.net/Philosophy#Release_early.2C_release_often
>>
>> And the BOM stuff is at:
>> *
>>
>> https://github.com/scijava/pom-scijava/blob/pom-scijava-8.3.0/pom.xml#L103-L819
>>
>> The downside, as you point out, of all components being release version
>> coupled is that it is annoying to have to do a "release cascade" to
>> propagate a bug fix from the lowest level components to the highest level
>> ones. We have some tooling to make that easier (I personally live in the
>> "releases should be as easy/automated/fast as possible" camp), but the
>> modularity does cost time sometimes. Hopefully a lot less time than
>> building your huge multi-module project from scratch every time, though!
>>
>> I also recently wrote a "melting pot" script to do end-to-end testing of
>> large component collections:
>>
>>
>> https://github.com/scijava/scijava-scripts/blob/d892adc0092c220ee1e597b9fb5a1fb067e4509b/melting-pot.sh
>>
>> This script builds and runs unit tests for all components of a large
>> collection at their respective versions, all in the same Java runtime, to
>> ensure that everything _really does_ work together at the versions you are
>> currently deploying to end users.
>>
>> I would be happy to know about other tooling people have created to help
>> with this sort of project structure.
>>
>> Regards,
>> Curtis
>>
>> On Tue, Sep 22, 2015 at 12:47 PM, Kevin Burton <burton@spinn3r.com>
>> wrote:
>>
>> We have a multi-module setup whereby we have about 150 independent
>>> modules.
>>>
>>> Our build takes a long time and actually slows down development as we
>>> have
>>> to do a compile of a LOT of source code to rebuild the project.
>>>
>>> Additionally, we have a lot of code that we want to Open Source.
>>>
>>> This has meant git submodules the IMO git submodules really don’t work
>>> when
>>> using branches.  They break and require a whole bunch of custom works and
>>> hack and when they DO break it’s confusing how to resolve them.
>>>
>>> This has meant that we’ve not really done a good job of OSSing our code
>>> base as its just too hard.
>>>
>>> What we’ve done to date is just have one major version number across all
>>> our projects.  So upgrading them and fixing their dependencies means
>>> that I
>>> just have to change a version number everywhere and I’m done.
>>>
>>> What I was thinking of is changing this strategy to use the maven
>>> "versions:use-latest-versions” plugin.
>>>
>>> What i would do is have a parent directory named ‘spinn3r’ which just
>>> has a
>>> bunch of git submodules.  We NEVER branch in this directory.
>>>
>>> It also means that any of our developers can check it out so that they
>>> have
>>> all of our source code.
>>>
>>> At this point I can use a normal development strategy for each project.
>>> They don’t use submodules which enables us to branch/merge easily.
>>>
>>> I can also have a dedicated IntelliJ or Eclipse project for each one and
>>> switch between them.
>>>
>>> Now the main issue I have is how do I bump releases easily and make sure
>>> all my code is using the latest version of its sibling projects.
>>>
>>> In the parent directory I can just run versions:use-latest-versions … on
>>> each one of the projects so that it automatically pulled in the latest
>>> version after a release.
>>>
>>> The only problem here is that there’s a dependency graph that needs to be
>>> considered.
>>>
>>> for example, if project A depends on project B, then we have to bump the
>>> version and push project B into maven before we upgrade dependencies on
>>> project A.
>>>
>>> This is a frustrating issue…
>>>
>>> --
>>>
>>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>>> Engineers!
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> <https://plus.google.com/102718274791889610666/posts>
>>>
>>>
>
> --
> Ron Wheeler
> President
> Artifact Software Inc
> email: rwheeler@artifact-software.com
> skype: ronaldmwheeler
> phone: 866-970-2435, ext 102
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message