maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harald Wellmann <hwellmann...@gmail.com>
Subject MNG-5896: Speed up builds by parallel POM downloads
Date Sat, 26 Dec 2015 10:39:09 GMT
When building a project with dependencies not yet available in the local 
repository, I noticed that Maven first downloads the dependency POMs 
sequentially and then proceeds downloading the dependency JARs with up 
to 5 threads in parallel, which is not optimal when the POMs are served 
by a high-latency repository manager.

There wasn't a lot of feedback on my enhancement request [1] or my 
original StackOverflow question [2], so I started digging into the 
source code and ended up with a patch [3].

The patch only affects Aether, not Maven Core, but since Aether is not 
the top-level project from the end user perspective and doesn't appear 
to be very active, I thought I'd better contact this mailing list first.

The basic idea of the patch is a clean separation of POM downloading 
from POM processing in DefaultDependencyCollector. Once these steps are 
separated, it is possible to download dependency POMs asynchronously and 
in parallel, while still processing the POM models sequentially to build 
the dependency graph in the correct order.

Since DefaultDependencyCollector holds a lot of global state and has 
rather long methods, I started by refactoring this class into smaller 
chunks to make the underlying logic more transparent. That's why the 
patch looks a bit large, but essentially it only affects a single 
original class.

I did a local build of Maven 3.4.0-SNAPSHOT using Aether 1.1.0-SNAPSHOT 
with my patch, with all tests passing.

I also ran maven-integration-testing on this patched Maven 
3.4.0-SNAPSHOT, with no new tests failing. (There is just one test which 
is broken since a recent change on trunk, see [4].)

Thanks for reading this far - it would be great if someone would take 
the time to look into the issue and the patch, and advise how to go on.


[1] https://issues.apache.org/jira/browse/MNG-5896
[2] 
http://stackoverflow.com/questions/32299902/parallel-downloads-of-maven-artifacts
[3] 
https://github.com/hwellmann/aether-core/commit/cdab4c40094ccf621370647f83ecda54684066ce
[4] 
https://builds.apache.org/job/maven-3.3-release-status-test-linux/lastCompletedBuild/testReport/

Regards,
Harald


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Mime
View raw message