maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tibor Digana (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SUREFIRE-1169) JUnit / Arquillian lifecycle friendly test execution with forkCount > 1 and reuseForks = true
Date Sun, 19 Jul 2015 23:33:05 GMT

    [ https://issues.apache.org/jira/browse/SUREFIRE-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632971#comment-14632971
] 

Tibor Digana commented on SUREFIRE-1169:
----------------------------------------

[~famod]
Please open issue on junit-team on GitHub regarding lazy loading of tests in Request class.
Regarding surefire fix we want to write extension (programming) API to customize surefire
behavior. Can you make any suggestion about the API? This is the branch with some starting
point of the extensions based on SPI
https://github.com/apache/maven-surefire/tree/s1

> JUnit / Arquillian lifecycle friendly test execution with forkCount > 1 and reuseForks
= true
> ---------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1169
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1169
>             Project: Maven Surefire
>          Issue Type: Improvement
>          Components: process forking
>    Affects Versions: 2.18.1
>            Reporter: Falko Modler
>         Attachments: SUREFIRE-1169_forksExecuteTestsOneByOne.patch
>
>
> The current approach to "stream" each test class name to a fork via {{TestProvidingInputStream}}
yields a good "load balancing" accross the forks but it *triggers the entire test lifecycle
for each test*.
> With {{forkCount = 1}}, all tests are executed in one set but with {{forkCount = n}}
(n > 1) each test is a separate "set" (so to say).
> This is very problematic in case you or a test framwork you are using relies on a certain
lifecycle.
> [Arquillian|http://arquillian.org/] for example ties various "events" to JUnit's lifecycle
methods, like {{AfterSuite}} to {{RunListener.testRunFinished(...)}} which triggers the shutdown
of the managed server etc.
> When using multiple forks, {{RunListener.testRunFinished(...)}} is called for *every*
single test class, triggering {{AfterSuite}} for every single test, although the fork will
receive further tests after that which should "reuse" the server.
> This is just an example. In fact the whole JUnit / Arquillian lifecycle is inconsistent
when using multiple forks.
> From a user perspective I wouldn't expect this behaviour:
> As {{forkCount = 1}} (and {{reusableForks = true}}) triggers {{RunListener.testRunFinished(...)}}
*once*, {{forkCount = n}} (and {{reusableForks = true}}) should trigger that method *n* times,
not *x* times.
> To be fair, the [documentation|https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html]
*does* contain a pointer to that problem by saying:
> {quote}
> When using reuseForks=true and a forkCount value larger than one, test classes are handed
over to the forked process one-by-one.
> {quote}
> But the consequences remain very unclear.
> *(Possible) Solution:*
> I took a stab at this and implemented an "eager test distribution" to the forks in {{ForkStart.java}}
and disabled streaming. Please see attached patch (to be applied against project root, 2.18.1).
> Patch Details:
> - New config property: {{ForkConfiguration.forksExecuteTestsOneByOne}}, set via Mojo
(default is true for backward compatibility, name is debatable)
> - When {{forksExecuteTestsOneByOne}} ist set to false, the {{messageQueue}} in {{ForkStarter.runSuitesForkOnceMultiple(...)}}
is *not* wrapped in fork specific {{TestProvidingInputStream}} instances to be consumed bit
by bit later on.
> Instead, the queue is consumed directly and each test class name is assigned to the respective
fork by creating a copy of the {{providerProperties}} which is filled individually for each
fork.
> E.g. for three forks and eight tests:
> -- fork 1 executes test 1, 4 and 7
> -- fork 2 executes test 2, 5 and 8
> -- fork 3 executes test 3 and 6
> - To have a clean {{providerProperties}} template I had to move {{DefaultScanResult.writeTo(...)}}
to the respective private methods. Otherwise the properties would have contained *all* tests
already.
> - I refactored some methods in {{ForkStarter}} to enhance readability and to reduce code
duplication.
> - The patch does *not* contain a test for the new behaviour but all existing tests passed.
> {{forksExecuteTestsOneByOne = false}} now leads to a consistent lifecycle.
> This solution has one downside: One or more forks could be overloaded while other forks
could "underloaded" because you cannot say how long each test will take. runOrder=balanced
could help here but has yet to be implemented for forks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message