maven-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Podgursky <bpodgur...@gmail.com>
Subject Re: Best strategy for using lots of modules/projects with some private and some OSS
Date Fri, 25 Sep 2015 17:06:46 GMT
Scanning a github org to generate builds would be pretty slick.  Definitely
interested in taking a look at that when it's released.

On Fri, Sep 25, 2015 at 5:20 AM, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:

> Aha! This is where
> https://github.com/jenkinsci/workflow-plugin/tree/master/multibranch
> and/or https://github.com/jenkinsci/literate-plugin/tree/master come
> into play... and shortly too there will be the multi-repo job type
> (which (wearing cloudbees hat) we are in the process of open sourcing
> (removes cloudbees hat)) which will scan all the repositories in your
> github/bitbucket organization and populate workflow-multibranch /
> literate projects for all repositories with the marker files.
>
> In short, you just point a mult-repo job type at GitHub, give it an
> API key and presto... jobs for all your projects and all the branches
> of all your project and all the PRs of all your projects with no
> maintenance at all and the full power of workflow to edit the job
> configuration using the Jenkinsfile at the root of each repo/branch
>
> On 25 September 2015 at 04:33, Ben Podgursky <bpodgursky@gmail.com> wrote:
> > Shouldn't require any additional CI servers -- we have one master with
> > hundreds of builds.  Depending on build frequency, you likely want a
> couple
> > slaves -- we have 7 but easily get by with 4.
> >
> > Setting up all the builds is some work but Jenkins will help you a lot.
> We
> > have a template project which we copy when setting up a new project, so
> > plugging in a new git repo and name takes less than a minute of work
> > (actually we have scripts which hit the jenkins API to create new
> projects,
> > but it's not critical).  And if you set up projects as maven builds, they
> > will detect downstream builds and trigger necessary rebuilds
> > automatically.  So Jenkins is usually doing less work overall, since one
> > monolithic project has to rebuild and re-rest * everything *, which would
> > take many hours.
> >
> > On Thu, Sep 24, 2015 at 8:09 PM, Kevin Burton <burton@spinn3r.com>
> wrote:
> >
> >> Pretty impressive.  I think another issue I hadn't thought of is the
> >> overhead of having a CI server/project for each project.
> >>
> >> If anything, you have to enumerate each project so you can see inside
> of it
> >> and what's happening.
> >>
> >> Having one macro one means you can see everything in one place but of
> >> course that has drawbacks too.
> >>
> >> On Tue, Sep 22, 2015 at 3:28 PM, Ben Podgursky <bpodgursky@gmail.com>
> >> wrote:
> >>
> >> > +1 for aggressive SNAPSHOT use
> >> >
> >> > Our setup.  YMMV:
> >> >
> >> > - We have ~75 independent projects and each of them is versioned and
> >> > deployed independently.  Few million lines of code.
> >> > - We have CI jenkins builds + unit test suites for every project, and
> we
> >> > deploy SNAPSHOT artifacts as soon as a build succeeds.
> >> > - Our OSS projects follow the same pattern (
> https://github.com/liveramp/
> >> ).
> >> > We do releases every so often for external users, but internally we
> use
> >> > snapshots which build for every new commit.
> >> > - Every internal dependency is on a SNAPSHOT version.
> >> > - All developers have all code checked out in intellij, source linked
> >> > - If you push changes to a library, you update all downstream
> usages.  No
> >> > exceptions.  Since you have all source in intellij, this usually isn't
> >> > hard.
> >> >
> >> > Our use-case is a bit different in that we don't really have
> "application
> >> > versions", since everything is back-end data pipeliens.  But when we
> >> > deploy, we build an artifact with locked SNAPSHOT versions which
> passes
> >> > integration tests and deploy that artifact.  We deploy many projects
> >> > several times a day.
> >> >
> >> > It takes a bit of diligence, but the payoff of not having to manage
> >> > internal versioning and have formal releases is huge.
> >> >
> >> > On Tue, Sep 22, 2015 at 1:14 PM, Ron Wheeler <
> >> > rwheeler@artifact-software.com
> >> > > wrote:
> >> >
> >> > > +1 (probably better and more complete than my description)
> >> > >
> >> > > Has anyone else looked at using an installer like izPack for
> assembling
> >> > > test setups?
> >> > > It integrates with Maven and will pick up all the right versions of
> >> jars
> >> > > and configuration files and build an installer that will drop the
> whole
> >> > set
> >> > > where ever you want to test.
> >> > > You can build an installer that will install on different OSs so you
> >> can
> >> > > take the same test jar and run it on Windows, Linux or a Mac and get
> >> the
> >> > > "right" configuration files for the target OS.
> >> > >
> >> > > Uses maven dependency management to get the jars from your repo and
> >> uses
> >> > a
> >> > > command language in XML to assemble the rest of your supporting data
> >> and
> >> > > configuration files.
> >> > >
> >> > > A bit smarter than Ant about building application run-time
> structures
> >> but
> >> > > that is all it does.
> >> > >
> >> > > Ron
> >> > >
> >> > >
> >> > > On 22/09/2015 3:43 PM, Curtis Rueden wrote:
> >> > >
> >> > >> Hi Kevin,
> >> > >>
> >> > >> My projects opt for independent versioning of modules to facilitate
> >> > >> "release early, release often." To do this for large sets of
> >> components
> >> > >> like yours requires a Bill of Materials -- i.e., common parent
POM
> >> with
> >> > >> dependencyManagement section.
> >> > >>
> >> > >> FWIW, the docs we have about our projects that work this way are
> at:
> >> > >> * http://imagej.net/Architecture
> >> > >>
> >> > >> And in particular:
> >> > >> * http://imagej.net/Architecture#Bill_of_Materials
> >> > >> *
> >> >
> http://imagej.net/Architecture#How_SciJava_achieves_reproducible_builds
> >> > >> * http://imagej.net/Philosophy#Release_early.2C_release_often
> >> > >>
> >> > >> And the BOM stuff is at:
> >> > >> *
> >> > >>
> >> > >>
> >> >
> >>
> https://github.com/scijava/pom-scijava/blob/pom-scijava-8.3.0/pom.xml#L103-L819
> >> > >>
> >> > >> The downside, as you point out, of all components being release
> >> version
> >> > >> coupled is that it is annoying to have to do a "release cascade"
to
> >> > >> propagate a bug fix from the lowest level components to the highest
> >> > level
> >> > >> ones. We have some tooling to make that easier (I personally live
> in
> >> the
> >> > >> "releases should be as easy/automated/fast as possible" camp),
but
> the
> >> > >> modularity does cost time sometimes. Hopefully a lot less time
than
> >> > >> building your huge multi-module project from scratch every time,
> >> though!
> >> > >>
> >> > >> I also recently wrote a "melting pot" script to do end-to-end
> testing
> >> of
> >> > >> large component collections:
> >> > >>
> >> > >>
> >> > >>
> >> >
> >>
> https://github.com/scijava/scijava-scripts/blob/d892adc0092c220ee1e597b9fb5a1fb067e4509b/melting-pot.sh
> >> > >>
> >> > >> This script builds and runs unit tests for all components of a
> large
> >> > >> collection at their respective versions, all in the same Java
> runtime,
> >> > to
> >> > >> ensure that everything _really does_ work together at the versions
> you
> >> > are
> >> > >> currently deploying to end users.
> >> > >>
> >> > >> I would be happy to know about other tooling people have created
to
> >> help
> >> > >> with this sort of project structure.
> >> > >>
> >> > >> Regards,
> >> > >> Curtis
> >> > >>
> >> > >> On Tue, Sep 22, 2015 at 12:47 PM, Kevin Burton <burton@spinn3r.com
> >
> >> > >> wrote:
> >> > >>
> >> > >> We have a multi-module setup whereby we have about 150 independent
> >> > >>> modules.
> >> > >>>
> >> > >>> Our build takes a long time and actually slows down development
> as we
> >> > >>> have
> >> > >>> to do a compile of a LOT of source code to rebuild the project.
> >> > >>>
> >> > >>> Additionally, we have a lot of code that we want to Open Source.
> >> > >>>
> >> > >>> This has meant git submodules the IMO git submodules really
don’t
> >> work
> >> > >>> when
> >> > >>> using branches.  They break and require a whole bunch of custom
> works
> >> > and
> >> > >>> hack and when they DO break it’s confusing how to resolve
them.
> >> > >>>
> >> > >>> This has meant that we’ve not really done a good job of
OSSing our
> >> code
> >> > >>> base as its just too hard.
> >> > >>>
> >> > >>> What we’ve done to date is just have one major version number
> across
> >> > all
> >> > >>> our projects.  So upgrading them and fixing their dependencies
> means
> >> > >>> that I
> >> > >>> just have to change a version number everywhere and I’m
done.
> >> > >>>
> >> > >>> What I was thinking of is changing this strategy to use the
maven
> >> > >>> "versions:use-latest-versions” plugin.
> >> > >>>
> >> > >>> What i would do is have a parent directory named ‘spinn3r’
which
> just
> >> > >>> has a
> >> > >>> bunch of git submodules.  We NEVER branch in this directory.
> >> > >>>
> >> > >>> It also means that any of our developers can check it out
so that
> >> they
> >> > >>> have
> >> > >>> all of our source code.
> >> > >>>
> >> > >>> At this point I can use a normal development strategy for
each
> >> project.
> >> > >>> They don’t use submodules which enables us to branch/merge
easily.
> >> > >>>
> >> > >>> I can also have a dedicated IntelliJ or Eclipse project for
each
> one
> >> > and
> >> > >>> switch between them.
> >> > >>>
> >> > >>> Now the main issue I have is how do I bump releases easily
and
> make
> >> > sure
> >> > >>> all my code is using the latest version of its sibling projects.
> >> > >>>
> >> > >>> In the parent directory I can just run
> versions:use-latest-versions …
> >> > on
> >> > >>> each one of the projects so that it automatically pulled in
the
> >> latest
> >> > >>> version after a release.
> >> > >>>
> >> > >>> The only problem here is that there’s a dependency graph
that
> needs
> >> to
> >> > be
> >> > >>> considered.
> >> > >>>
> >> > >>> for example, if project A depends on project B, then we have
to
> bump
> >> > the
> >> > >>> version and push project B into maven before we upgrade
> dependencies
> >> on
> >> > >>> project A.
> >> > >>>
> >> > >>> This is a frustrating issue…
> >> > >>>
> >> > >>> --
> >> > >>>
> >> > >>> We’re hiring if you know of any awesome Java Devops or Linux
> >> Operations
> >> > >>> Engineers!
> >> > >>>
> >> > >>> Founder/CEO Spinn3r.com
> >> > >>> Location: *San Francisco, CA*
> >> > >>> blog: http://burtonator.wordpress.com
> >> > >>> … or check out my Google+ profile
> >> > >>> <https://plus.google.com/102718274791889610666/posts>
> >> > >>>
> >> > >>>
> >> > >
> >> > > --
> >> > > Ron Wheeler
> >> > > President
> >> > > Artifact Software Inc
> >> > > email: rwheeler@artifact-software.com
> >> > > skype: ronaldmwheeler
> >> > > phone: 866-970-2435, ext 102
> >> > >
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> >> > > For additional commands, e-mail: users-help@maven.apache.org
> >> > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >>
> >> We’re hiring if you know of any awesome Java Devops or Linux Operations
> >> Engineers!
> >>
> >> Founder/CEO Spinn3r.com
> >> Location: *San Francisco, CA*
> >> blog: http://burtonator.wordpress.com
> >> … or check out my Google+ profile
> >> <https://plus.google.com/102718274791889610666/posts>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message