maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milos Kleint <mkle...@gmail.com>
Subject Re: MNG-5896: Speed up builds by parallel POM downloads
Date Sun, 27 Dec 2015 21:56:58 GMT
Nice work! Looking forward to have this integrated in maven codebase.

We've solved the same problem internally by developing a service + maven
extension that downloads the maven pom files in bulk (one or more zips). We
are using bamboo aws elastic agents a lot and on a clean local repository,
this can save a minute or two for large builds (when your nexus proxy is
also placed in aws as well, when connecting to central even more).

Milos

On Sat, Dec 26, 2015 at 9:39 PM, Harald Wellmann <hwellmann.de@gmail.com>
wrote:

> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven first downloads the dependency POMs
> sequentially and then proceeds downloading the dependency JARs with up to 5
> threads in parallel, which is not optimal when the POMs are served by a
> high-latency repository manager.
>
> There wasn't a lot of feedback on my enhancement request [1] or my
> original StackOverflow question [2], so I started digging into the source
> code and ended up with a patch [3].
>
> The patch only affects Aether, not Maven Core, but since Aether is not the
> top-level project from the end user perspective and doesn't appear to be
> very active, I thought I'd better contact this mailing list first.
>
> The basic idea of the patch is a clean separation of POM downloading from
> POM processing in DefaultDependencyCollector. Once these steps are
> separated, it is possible to download dependency POMs asynchronously and in
> parallel, while still processing the POM models sequentially to build the
> dependency graph in the correct order.
>
> Since DefaultDependencyCollector holds a lot of global state and has
> rather long methods, I started by refactoring this class into smaller
> chunks to make the underlying logic more transparent. That's why the patch
> looks a bit large, but essentially it only affects a single original class.
>
> I did a local build of Maven 3.4.0-SNAPSHOT using Aether 1.1.0-SNAPSHOT
> with my patch, with all tests passing.
>
> I also ran maven-integration-testing on this patched Maven 3.4.0-SNAPSHOT,
> with no new tests failing. (There is just one test which is broken since a
> recent change on trunk, see [4].)
>
> Thanks for reading this far - it would be great if someone would take the
> time to look into the issue and the patch, and advise how to go on.
>
>
> [1] https://issues.apache.org/jira/browse/MNG-5896
> [2]
> http://stackoverflow.com/questions/32299902/parallel-downloads-of-maven-artifacts
> [3]
> https://github.com/hwellmann/aether-core/commit/cdab4c40094ccf621370647f83ecda54684066ce
> [4]
> https://builds.apache.org/job/maven-3.3-release-status-test-linux/lastCompletedBuild/testReport/
>
> Regards,
> Harald
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message