maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martijn Dashorst <martijn.dasho...@gmail.com>
Subject Re: Summary of meeting about Maven performance improvements
Date Wed, 24 Apr 2019 19:19:47 GMT
There's this umbrella of projects called Wicket Stuff at github.

https://github.com/wicketstuff/core

It has a typical long build time, but not a very complicated setup
(e.g. the depth of nested projects is about 3 max)

You might consider using this as a public benchmark.

Looking forward to improvements in build times.

Martijn

On Wed, Apr 24, 2019 at 8:30 PM Arnaud Héritier <aheritier@gmail.com> wrote:
>
> Great findings then!
> It’s not easy to catch and my experience is that such big projects are
> often in corporate environments which aren’t open at all and not really
> working with oss projects thus it’s not surprising to have this surfacing
> on Gradle support side.
>
> Le mer. 24 avr. 2019 à 19:09, Karl Heinz Marbaise <khmarbaise@gmx.de> a
> écrit :
>
> > Hi,
> >
> > On 24.04.19 17:52, Robert Scholte wrote:
> > > On Wed, 24 Apr 2019 12:45:42 +0200, Arnaud Héritier
> > > <aheritier@gmail.com> wrote:
> > >
> > >> Just for my knowledge, is it a regression in a recent version or it is
> > >> like
> > >> this for ages ?
> > >
> > > It is like this for ages. AFAIK nobody of the team monitored memory
> > > consumption, focus was on correct + "readable" code.
> >
> > I had done some investigations a longer time ago...related to MNG-6030
> > https://github.com/khmarbaise/maven-test-project-generator
> >
> > and yes it's not a regression...
> >
> > Kind regards
> > Karl Heinz
> >
> > >
> > > Robert
> > >
> > >>
> > >>
> > >> Le mer. 24 avr. 2019 à 10:56, Benedikt Ritter <britter@apache.org>
a
> > >> écrit :
> > >>
> > >>> Am Mi., 24. Apr. 2019 um 10:50 Uhr schrieb Benedikt Ritter <
> > >>> britter@apache.org>:
> > >>>
> > >>> > Hello,
> > >>> >
> > >>> > this is a summary of a video conference call that happened yesterday
> > >>> > (April 24).
> > >>> >
> > >>>
> > >>> Sorry, actually yesterday was April 23... :o)
> > >>>
> > >>>
> > >>> >
> > >>> > Topic:
> > >>> > Discussion about performance improvements that have been proposed
by
> > >>> > Stefan Oehme, namely:
> > >>> >
> > >>> > - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource (
> > >>> > https://github.com/apache/maven/pull/244)
> > >>> > - [MNG-6633] - Reduce memory usage of excludes (
> > >>> > https://github.com/apache/maven/pull/243) - Speed up project
> > >>> discovery (
> > >>> > https://github.com/apache/maven/pull/242) - Make location handling
> > >>> more
> > >>> > memory efficient (https://github.com/codehaus-plexus/modello/pull/31
> > )
> > >>> >
> > >>> > The goal of this call was to give some more insights into how
Stefan
> > >>> found
> > >>> > the improvements and to better understand what is missing before
> > these
> > >>> > changes be merged.
> > >>> >
> > >>> > Attendees of the call:
> > >>> > - Benedikt Ritter (Gradle Inc.)
> > >>> > - Stefan Oehme (Gradle Inc.)
> > >>> > - Robert Scholte (Apache Maven Team)
> > >>> > - Hervé Boutemy (Apache Maven Team; joined about half an hour
after
> > >>> the
> > >>> > call started)
> > >>> >
> > >>> > Summary:
> > >>> >
> > >>> > Stefan gave some insights into how he discovered bottlenecks in
> > Maven:
> > >>> >
> > >>> >    -
> > >>> >
> > >>> >    One of our customers has a huge Maven build:
> > >>> >    -
> > >>> >
> > >>> >       Lots of sub projects (2000)
> > >>> >       -
> > >>> >
> > >>> >       Lots of entries in dependency management (4000)
> > >>> >       -
> > >>> >
> > >>> >       Results in a lot of garbage collection
> > >>> >       -
> > >>> >
> > >>> >    Problems discovered in that build:
> > >>> >    -
> > >>> >
> > >>> >       Re-parsing project POMs during dependency resolution
> > >>> >       -
> > >>> >
> > >>> >       Model objects are too large because of location tracking
> > >>> >       -
> > >>> >
> > >>> >       Low-level bottlenecks in project discovery (especially version
> > >>> >       parsing)
> > >>> >       -
> > >>> >
> > >>> >    Customer now has a Maven fork with the proposed changes included:
> > >>> >    -
> > >>> >
> > >>> >       1h 50min, 12GB RAM without changes
> > >>> >       -
> > >>> >
> > >>> >       45min, 8GB RAM with changes
> > >>> >
> > >>> >
> > >>> > Robert:
> > >>> >
> > >>> >    -
> > >>> >
> > >>> >    How to ensure that improvements are not broken?
> > >>> >    -
> > >>> >
> > >>> >    No answer to how to test this
> > >>> >
> > >>> >
> > >>> > Stefan gave some insights into how performance testing works in
the
> > >>> Gradle
> > >>> > project:
> > >>> >
> > >>> >    -
> > >>> >
> > >>> >    Build has a project generator
> > >>> >    -
> > >>> >
> > >>> >    Create different projects in different shapes (e.g. lots of
> > >>> >    subprojects, deeply nested projects) during the build
> > >>> >    -
> > >>> >
> > >>> >    Download old Gradle version and run the build on generated
> > projects
> > >>> >    -
> > >>> >
> > >>> >    Run build again with current Gradle version
> > >>> >    -
> > >>> >
> > >>> >    Compare results
> > >>> >    -
> > >>> >
> > >>> >    use statistic methods to filter out variance
> > >>> >    -
> > >>> >
> > >>> >    Downside to this approach is that it requires a lot of computing
> > >>> >    resources
> > >>> >
> > >>> > More information can be found on GitHub:
> > >>> > https://github.com/gradle/gradle/tree/master/subprojects/performance
> > >>> > The corresponding TeamCity build can be found here:
> > >>> >
> > >>>
> > https://builds.gradle.org/viewLog.html?buildId=22179604&buildTypeId=Gradle_Check_PerformanceExperimentCoordinator&tab=report_project941_Performance&branch_Gradle_Check_Stage_ReadyforRelease=master
> > >>>
> > >>> (use
> > >>> > "Login as guest" to view)
> > >>> >
> > >>> > Robert:
> > >>> >
> > >>> >    -
> > >>> >
> > >>> >    What about measuring performance using instruction calls?
> > >>> >
> > >>> >
> > >>> > Stefan:
> > >>> >
> > >>> >    -
> > >>> >
> > >>> >    The performance improvements we found were mostly about garbage
> > >>> being
> > >>> >    created
> > >>> >    -
> > >>> >
> > >>> >    Measuring using instruction calls is interesting
> > >>> >    -
> > >>> >
> > >>> >    ... but it is also very machine dependent
> > >>> >
> > >>> >
> > >>> > Robert:
> > >>> >
> > >>> >    -
> > >>> >
> > >>> >    We need to find out who is interested in these kind improvements
> > >>> >    inside the Maven community.
> > >>> >    -
> > >>> >
> > >>> >    Build a community of people who would like to work on these
kind
> > of
> > >>> >    things.
> > >>> >
> > >>> >
> > >>> > Stefan:
> > >>> >
> > >>> >    -
> > >>> >
> > >>> >    It's easy to get started. We just used open source tools:
> > >>> >    -
> > >>> >
> > >>> >    We used async-profiler for measuring things (
> > >>> >    https://github.com/jvm-profiling-tools/async-profiler)
> > >>> >    -
> > >>> >
> > >>> >    Heap dumps for analyzing memory usage
> > >>> >
> > >>> > To get started with performance tests in the maven project:
> > >>> >
> > >>> >    -
> > >>> >
> > >>> >    Start with only a few test projects
> > >>> >    -
> > >>> >
> > >>> >    The Gradle generator is Apache License v2 and can be used as
a
> > >>> >    starting point to generate a big maven project
> > >>> >
> > >>> >
> > >>> > Hervé:
> > >>> >
> > >>> >    -
> > >>> >
> > >>> >    PRs should be merged soon
> > >>> >    -
> > >>> >
> > >>> >    Discussion need to be resolved
> > >>> >    -
> > >>> >
> > >>> >    Why was the PR not merged after the discussion and resolving
all
> > >>> >    issues with the code?
> > >>> >    -
> > >>> >
> > >>> >    Hervé will take care that the changes are merged soon
> > >>> >
> > >>> >
> > >>> > Thank you!
> > >>> > Benedikt
> > >>> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> > --
> -----
> Arnaud Héritier
> http://aheritier.net
> Mail/GTalk: aheritier AT gmail DOT com
> Twitter/Skype : aheritier



-- 
Become a Wicket expert, learn from the best: http://wicketinaction.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Mime
View raw message