lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Høydahl <>
Subject Re: Gradle build effort (respin, please read)
Date Tue, 03 Dec 2019 09:48:18 GMT
Great work Dawid. Just trying out solr tests now.
I see that it is not very smart in parallel scheduling. But that is probably something we
can improve over time?
Is there some env.var or global setting to force e.g. -Ptests.seed=deadbeef for faster runs?
We probably don’t need to randomize everything everywhere every time?

Also, I like your moving of build files from the source tree. Will soon get used to gradlew
-p solr/packaging assemble.
Wonder how this will work with Admin UI changes. Hopefully you can just run a new assemble
and reload browser on each change.


> 3. des. 2019 kl. 09:35 skrev Dawid Weiss <>:
>> gradlew -p lucene test
>> That took my machine 27 minutes!  I can see it used like 7 threads when I'd rather
it use 3-4.  That's probably why.
> I capped the execution of parallel tests to this
> (gradle/testing/defaults-tests.gradle):
>      // Set up default parallel execution limits.
>      maxParallelForks = (int) Math.max(1,
> Math.min(Runtime.runtime.availableProcessors() / 2.0, 3.0))
> but this limit applies per-project and gradle runs with N workers in
> parallel mode (where N equals the number of cores). For example, on my
> 8-core Linux machine (SSD drive) the test time for
> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894
> is 11m 32s. If I limit max workers to 4:
> ./gradlew -p lucene cleanTest test -Ptests.seed=28BC12BA7C2CD894 --max-workers=4
> the build time is XXX.
> It is obvious we need to fine-tune this but it's not obvious what the
> defaults should be because there are many dimensions to performance:
> I/O congestion, CPU cores, overall memory bandwidth and operating
> system (underlying filesystem implementation mostly, I believe). Also,
> as Mark pointed out before, the test runner is slightly different in
> Gradle and doesn't load-balance tests too efficiently (no work
> stealing).
> Finally, all the above also doesn't mean we can only think of
> improving gradle performance and not tests themselves... Some of them
> are just (very) slow. :)
> If you're willing to experiment then try to run with a varying number
> of workers (and identical seed to keep the tests the same). Then see
> which one turns out to be the best for you (and report back). I think
> it'll be half the number of cores (effectively physical cores) unless
> you have a very beefy machine when memory bandwidth and I/O comes into
> play.
> gradlew --max-workers N -Ptests.seed=deadbeef
>> gradlew :helpAnt
>> This failed: --
> Corrected, apologies.
> Dawid
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message