www-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bayer <andrew.ba...@gmail.com>
Subject Re: builds.apache.org Jenkins status as of end of day May 27
Date Wed, 28 May 2014 17:38:09 GMT
There are a ton of issues over at the Jenkins JIRA relating to
problems with the Maven plugin - they show up particularly at scale.
The problem is that a MavenModuleSet (the class for the overall Maven
project) contains a bunch of MavenModules, and both of those inherit
from AbstractProject. So effectively, each MavenModuleSet project is a
bunch of separate jobs when it comes to configuration, builds, etc -
this creates a *lot* of inefficiencies in terms of loading, memory,
etc. For example, there are 1100 or so jobs in builds.apache.org,
which puts it in the top 40 or so of publicly reporting Jenkins
masters. But 600+ of those jobs are MavenModuleSets with an average of
nearly 40 modules per project, resulting in another 23000 or so
MavenModules - so at startup and whenever anything's processing across
all jobs, we're not actually acting on 1100 jobs, we're acting on 24k
jobs.

There are some other particularly nasty problems - the Maven
interceptor used for MavenModuleSet jobs has a history of hitting bugs
in the remoting architecture that freestyle projects never touch,
build health and some other metrics that look at the last X builds get
really, really ugly with MavenModuleSets due to them having to load
every build of every module where a freestyle project only needs to
look at X builds, plus issues I haven't been able to thoroughly
unravel in the lazy loading of builds not actually working right with
MavenModuleSets (so that they end up not really doing lazy loading and
in fact loading everything into memory, etc).

The primary Jenkins master at my work has 1600 jobs and 90k plus
builds, in the same ballpark as what we'd see on builds.apache.org if
we weren't using MavenModuleSets, and its startup time is 5 or so
minutes, vs 20 minutes or so for builds.apache.org, and our internal
master's memory usage is dramatically lower as well - MavenModuleSets
are just brutally inefficient and problematic for performance at
scale.

A.

On Wed, May 28, 2014 at 5:12 AM, Andreas Veithen
<andreas.veithen@gmail.com> wrote:
> On Tue, May 27, 2014 at 11:41 PM, Andrew Bayer <andrew.bayer@gmail.com> wrote:
>> I disabled this due to the huge pains we've had that are caused by
>> various flaws/oversights in the MavenModuleSet project type's
>> architecture.
>
> Can you provide evidence for that claim? We've been using Maven
> projects on Jenkins for a long time. There certainly have been issues
> linked to that particular type, but that is of course expected given
> the functionality that the plugin adds. I'm surprised that all of a
> sudden claims are made that that plugin/project type is crap.
>
> Andreas

Mime
View raw message