maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Heinz Marbaise <khmarba...@gmx.de>
Subject Re: Multi-threaded deploy at end
Date Fri, 08 May 2015 20:06:50 GMT
Hi Jefferey,

On 5/8/15 5:37 PM, Jeffrey E Care wrote:
> Karl Heinz Marbaise <khmarbaise@gmx.de> wrote on 05/07/2015 02:04:51 PM:
>
>> On 5/7/15 7:57 PM, Jeffrey E Care wrote:
>>> We have several medium-to-large multi-module builds. We use deploy at
> end
>>> and I've noticed that it's taking a long time to run the deploys. I
> dug
>>> into the deploy plugin & noticed that for deploy at end the
> deployments
>>> are run serially. Has there been any thought to using multiple threads
> for
>>> this use case? Would you be open to a contribution implementing this?
>>
>> any contriubtion is really appreciated...sounds extremly interesting,
>> cause at the moment i'm running a large build with 320 modules which
>> took about 2 minutes to upload those artifacts to nexus...
>
> 2 minutes for 320 modules? I wonder if there's a problem with our nexus.
> In my largest build we're taking 13 minutes to upload 103 modules...
>

So i have rechecked the numbers...

number of modules: 311
transfered data to Nexus: 1.34 GiB
upload time: 2 Minutes  (approxm. 11 MiB/s)
total number of tests: 4855
run time of tests: 390 seconds
total build time: 11 minutes

The number of modules can simply be calculated by using the log file 
(Maven log output for example on Jenkins) via this:

1. Number of modules

cat log | egrep -a "\[INFO\]\ .* SUCCESS \[.*\]" | wc -l

2. Uplodated size

cat log | egrep -a "\[INFO\]\ Uploaded\: " | cut -d"(" -f2 | cut -d" " 
-f1-2 |  sed -e 's/\ KB/*1024/' | sed -e 's/\ B//' | paste -sd+ | bc -l

3. Number of uploaded artifacts:

cat log | grep -a "^\[INFO\] Uploaded: "  | grep -v 
".*/maven-metadata.xml .*" | wc -l

4. number of tests:

log | egrep -a "^Tests run: [0-9]+, Failures: [0-9]+, Errors: [0-9]+, 
Skipped: [0-9]+, Time elapsed: " | cut -d" " -f 3| cut -d"," -f1 | paste 
-sd+ | bc

5. runtime of tests:

cat log | egrep -a "^Tests run: [0-9]+, Failures: [0-9]+, Errors: 
[0-9]+, Skipped: [0-9]+, Time elapsed: " | cut -d":" -f 6 | cut -d" " 
-f2 | paste -sd+ | bc -l


Do you know the total size of the artifacts which are uploaded ?
Can you analyze how many artiacts being uploaded ?
How long does your build take?
How many tests do you have?
How long take the tests to run?

Do you know the number of lines of code (measured by SonarQube?) ( I 
have about 439,000)...

Before i have started to optimize the build process i had 8-12 minutes 
upload time to nexus (with less modules)...and build times at about 45 
minutes...

Based on this numbers we can make our builds comparable...

Kind regards
Karl Heinz Marbaise

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Mime
View raw message