www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Veithen <andreas.veit...@gmail.com>
Subject Re: builds.apache.org Jenkins status as of end of day May 27
Date Sun, 01 Jun 2014 10:21:10 GMT
See comments inline.



On Wed, May 28, 2014 at 6:38 PM, Andrew Bayer <andrew.bayer@gmail.com> wrote:
> There are a ton of issues over at the Jenkins JIRA relating to
> problems with the Maven plugin - they show up particularly at scale.
> The problem is that a MavenModuleSet (the class for the overall Maven
> project) contains a bunch of MavenModules, and both of those inherit
> from AbstractProject. So effectively, each MavenModuleSet project is a
> bunch of separate jobs when it comes to configuration, builds, etc -
> this creates a *lot* of inefficiencies in terms of loading, memory,
> etc. For example, there are 1100 or so jobs in builds.apache.org,
> which puts it in the top 40 or so of publicly reporting Jenkins
> masters. But 600+ of those jobs are MavenModuleSets with an average of
> nearly 40 modules per project, resulting in another 23000 or so
> MavenModules - so at startup and whenever anything's processing across
> all jobs, we're not actually acting on 1100 jobs, we're acting on 24k
> jobs.

Note that with 600+ Maven jobs, you will likely many people unhappy if
you replace them by freestyle projects and they no longer get the
features they were used to. So this needs to be considered very

> There are some other particularly nasty problems - the Maven
> interceptor used for MavenModuleSet jobs has a history of hitting bugs
> in the remoting architecture that freestyle projects never touch,
> build health and some other metrics that look at the last X builds get
> really, really ugly with MavenModuleSets due to them having to load
> every build of every module where a freestyle project only needs to
> look at X builds, plus issues I haven't been able to thoroughly
> unravel in the lazy loading of builds not actually working right with
> MavenModuleSets (so that they end up not really doing lazy loading and
> in fact loading everything into memory, etc).

If the problem is caused by a large number of per-module builds, then
we should probably look at the following issue first. Take a random
Maven job and look at the builds for one of the modules:


You will notice that the list contains old builds from December 2013
and January 2014 that for some reasons are not cleaned up. If you are
right and the problem is caused by a large number of per-module
builds, then cleaning up the old builds would be the first step.

> The primary Jenkins master at my work has 1600 jobs and 90k plus
> builds, in the same ballpark as what we'd see on builds.apache.org if
> we weren't using MavenModuleSets, and its startup time is 5 or so
> minutes, vs 20 minutes or so for builds.apache.org,

That is a factor of 4, but you are comparing completely different
configurations. Do you have any direct evidence that the startup time
is dominated by the number of jobs or builds?

> and our internal
> master's memory usage is dramatically lower as well

What amount of memory are we talking about here?

> - MavenModuleSets
> are just brutally inefficient and problematic for performance at
> scale.
> A.
> On Wed, May 28, 2014 at 5:12 AM, Andreas Veithen
> <andreas.veithen@gmail.com> wrote:
>> On Tue, May 27, 2014 at 11:41 PM, Andrew Bayer <andrew.bayer@gmail.com> wrote:
>>> I disabled this due to the huge pains we've had that are caused by
>>> various flaws/oversights in the MavenModuleSet project type's
>>> architecture.
>> Can you provide evidence for that claim? We've been using Maven
>> projects on Jenkins for a long time. There certainly have been issues
>> linked to that particular type, but that is of course expected given
>> the functionality that the plugin adds. I'm surprised that all of a
>> sudden claims are made that that plugin/project type is crap.
>> Andreas

View raw message