maven-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: [surefire] Combining test, runOrder and forkCount
Date Fri, 30 Aug 2013 10:55:18 GMT
Thanks Andreas,

Removing parallel=classes did the trick for timely console reporting.
Having parallel=classes didn't bother m-surefire-p in 2.15 and earlier.
With forkCount > 1 and reuseForks = false, parallel=classes should just be
ignored. But, as long as there is solution, I'm fine.

IMO specifying test parameter shouldn't disable the ordering (runOrder),
since specifying test parameter (by docs) is equivalent to specifying
includes patterns in pom, and includes should be used only to filter
scanned test classes, ordering should be applied after scanning, and before
executing test classes.

I understand that there are no guarantees on order when only two test
classes are run in parallel with forkCount > 1.
Still if runOrder is applied before forking VMs and forked VMs as you
describe poll for next test class to execute, and I run a dozen of tests
(specified with test parameter) each of which takes different number of
more than few seconds to finish, and I have forkCount set to 2, I'd expect
runOrder to be somewhat respected - for the first two tests which get run
it would be random, but for the rest it should be pretty much in specified
runOrder, until two finish at roughly same time and there is (again as on
beginning) concurrency which test class will run next. Do you agree?
I tried that (test specifying dozen of test classes, forkCount at 2,
reuseForks at false, and different runOrder settings), and still I don't
see runOrder being respected.

Btw my original suspicion, that the two of Apache Mahout tests do not run
well together, was wrong - failing test had a bug, it makes use of random
number generator, but didn't set test seed to make execution deterministic.

Kind regards,
Stevo Slavic.


On Fri, Aug 30, 2013 at 8:06 AM, Andreas Gudian <andreas.gudian@gmail.com>wrote:

> I'll have to check the code, but I'm pretty sure that the order of the test
> classes to be handed over to the forked VMs is supposed to be what was is
> specified with runOrder.
>
> However, using -Dtest= might disable the ordering - but again, I'd have to
> check the code.
>
> Bit there's more: when having two classes to test and two forked VMs, there
> is no real way to tell which test comes first. Both VMs are started right
> away and only when the are up and running and ready to execute a test, they
> ask the main process for one test class. Which VM will be the first to get
> a class is quite random.
>
> For the delayed reporting to console, try removing parallel=classes from
> the config. IThe classes-setting has no effect when forking to more than
> one fork anyway. If the output is still delayed then, create a bug-report,
> please.
>
>
> Am Donnerstag, 29. August 2013 schrieb Stevo Slavić :
>
> > Hello Apache Maven community,
> >
> > To reproduce random failing test issue, I'm trying to run two tests in
> > parallel, but start running them in specific order. Problem is that
> > runOrder (added via SUREFIRE-614) doesn't seem to be respected, at least
> > not when forkCount and/or test is specified as well.
> >
> > When trying to run two tests from Apache Mahout core module (trunk), I
> > cannot get surefire to run them in reverse alphabetical order.
> >
> > Regardless if I run:
> > mvn -Dtest=*KMeansTest -DrunOrder=reversealphabetical test
> >
> > or
> > mvn -Dtest=*KMeansTest -DrunOrder=alphabetical test
> >
> > the two tests which match the pattern always get run in alphabetical
> order:
> >
> > -------------------------------------------------------
> >  T E S T S
> > -------------------------------------------------------
> > Running org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > Running
> org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.706
> sec
> > - in org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.15
> sec -
> > in org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> >
> > Results :
> >
> > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> >
> >
> >
> >
> > Surefire 2.15 is used, and configuration is:
> >
> >
> >       <plugin>
> >         <groupId>org.apache.maven.plugins</groupId>
> >         <artifactId>maven-surefire-plugin</artifactId>
> >         <configuration>
> >           <forkCount>1.5C</forkCount>
> >           <reuseForks>false</reuseForks>
> >           <threadCount>1</threadCount>
> >           <perCoreThreadCount>false</perCoreThreadCount>
> >           <parallel>classes</parallel>
> >           <argLine>-Xmx512m</argLine>
> >           <testFailureIgnore>false</testFailureIgnore>
> >           <redirectTestOutputToFile>true</redirectTestOutputToFile>
> >           <systemPropertyVariables>
> >
> > <mahout.test.directory>${project.build.directory}</mahout.test.directory>
> >           </systemPropertyVariables>
> >         </configuration>
> >       </plugin>
> >
> >
> > When I use latest surefire 2.16, outcome is even more odd - output that
> any
> > of the tests is running takes longer to render, it seems to be rendered
> > together with info that test has completed, so by looking at build log
> > output it may appear as if tests were not running in parallel even though
> > they did:
> >
> > -------------------------------------------------------
> >  T E S T S
> > -------------------------------------------------------
> > Running org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.347
> sec
> > - in org.apache.mahout.clustering.streaming.cluster.BallKMeansTest
> > Running
> org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.766
> sec
> > - in org.apache.mahout.clustering.streaming.cluster.StreamingKMeansTest
> >
> > Results :
> >
> > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> >
> >
> > Here is my "mvn --version" output:
> >
> > [sslavic@laptop core]$ mvn --version
> > Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-28
> > 04:15:32+0200)
> > Maven home: /home/sslavic/work/tools/apache-maven
> > Java version: 1.7.0_25, vendor: Oracle Corporation
> > Java home: /usr/java/jdk1.7.0_25/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "linux", version: "3.10.9-200.fc19.x86_64", arch: "amd64",
> family:
> > "unix"
> >
> >
> >
> > Am I doing something wrong or is this behavior bug (or two bugs) in
> > surefire?
> >
> > Kind regards,
> > Stevo Slavic.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message