maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tibor Digana (JIRA)" <>
Subject [jira] [Commented] (SUREFIRE-1169) JUnit / Arquillian lifecycle friendly test execution with forkCount > 1 and reuseForks = true
Date Fri, 17 Jul 2015 14:40:06 GMT


Tibor Digana commented on SUREFIRE-1169:

Maybe not necessary to make too big changes.
You can see JUnit4Provicer and JUnitCoreProvider and wrapper.

AFAIK all you want the surefire plugin to call testRunStarted and testRunFinished called just
once per JVM.
If Arquillian specifies system property, -Dlistener=JUnitListener, then it is just a question
to suppress multiple calls on custom listeners. Right?

Can we enable just only one call of  testRunStarted and testRunFinished on custom listeners
per JVM?
Would it have any negative side effect on surefire functionality?

> JUnit / Arquillian lifecycle friendly test execution with forkCount > 1 and reuseForks
= true
> ---------------------------------------------------------------------------------------------
>                 Key: SUREFIRE-1169
>                 URL:
>             Project: Maven Surefire
>          Issue Type: Improvement
>          Components: process forking
>    Affects Versions: 2.18.1
>            Reporter: Falko Modler
>         Attachments: SUREFIRE-1169_forksExecuteTestsOneByOne.patch
> The current approach to "stream" each test class name to a fork via {{TestProvidingInputStream}}
yields a good "load balancing" accross the forks but it *triggers the entire test lifecycle
for each test*.
> With {{forkCount = 1}}, all tests are executed in one set but with {{forkCount = n}}
(n > 1) each test is a separate "set" (so to say).
> This is very problematic in case you or a test framwork you are using relies on a certain
> [Arquillian|] for example ties various "events" to JUnit's lifecycle
methods, like {{AfterSuite}} to {{RunListener.testRunFinished(...)}} which triggers the shutdown
of the managed server etc.
> When using multiple forks, {{RunListener.testRunFinished(...)}} is called for *every*
single test class, triggering {{AfterSuite}} for every single test, although the fork will
receive further tests after that which should "reuse" the server.
> This is just an example. In fact the whole JUnit / Arquillian lifecycle is inconsistent
when using multiple forks.
> From a user perspective I wouldn't expect this behaviour:
> As {{forkCount = 1}} (and {{reusableForks = true}}) triggers {{RunListener.testRunFinished(...)}}
*once*, {{forkCount = n}} (and {{reusableForks = true}}) should trigger that method *n* times,
not *x* times.
> To be fair, the [documentation|]
*does* contain a pointer to that problem by saying:
> {quote}
> When using reuseForks=true and a forkCount value larger than one, test classes are handed
over to the forked process one-by-one.
> {quote}
> But the consequences remain very unclear.
> *(Possible) Solution:*
> I took a stab at this and implemented an "eager test distribution" to the forks in {{}}
and disabled streaming. Please see attached patch (to be applied against project root, 2.18.1).
> Patch Details:
> - New config property: {{ForkConfiguration.forksExecuteTestsOneByOne}}, set via Mojo
(default is true for backward compatibility, name is debatable)
> - When {{forksExecuteTestsOneByOne}} ist set to false, the {{messageQueue}} in {{ForkStarter.runSuitesForkOnceMultiple(...)}}
is *not* wrapped in fork specific {{TestProvidingInputStream}} instances to be consumed bit
by bit later on.
> Instead, the queue is consumed directly and each test class name is assigned to the respective
fork by creating a copy of the {{providerProperties}} which is filled individually for each
> E.g. for three forks and eight tests:
> -- fork 1 executes test 1, 4 and 7
> -- fork 2 executes test 2, 5 and 8
> -- fork 3 executes test 3 and 6
> - To have a clean {{providerProperties}} template I had to move {{DefaultScanResult.writeTo(...)}}
to the respective private methods. Otherwise the properties would have contained *all* tests
> - I refactored some methods in {{ForkStarter}} to enhance readability and to reduce code
> - The patch does *not* contain a test for the new behaviour but all existing tests passed.
> {{forksExecuteTestsOneByOne = false}} now leads to a consistent lifecycle.
> This solution has one downside: One or more forks could be overloaded while other forks
could "underloaded" because you cannot say how long each test will take. runOrder=balanced
could help here but has yet to be implemented for forks.

This message was sent by Atlassian JIRA

View raw message