lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dawid Weiss (Commented) (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3492) Extract a generic framework for running randomized tests.
Date Fri, 21 Oct 2011 19:46:32 GMT


Dawid Weiss commented on LUCENE-3492:

In response to my question whether the idea of randomized testing is new Yuriy Pasichnyk passed
me the info about Haskell's QuickCheck project. Indeed, the idea is pretty much the same (with
differences concerning implementation details, not the concept itself).

There is a Java port of this too, if you check out Wikipedia. The implementation follows a
different direction compared to what I implemented, but there are also pieces that are nearly
1:1 identical copies. Good to know -- this means I wasn't completely wrong in my goals.
> Extract a generic framework for running randomized tests.
> ---------------------------------------------------------
>                 Key: LUCENE-3492
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Minor
>             Fix For: 4.0
>         Attachments: Screen Shot 2011-10-06 at 12.58.02 PM.png
> {color:red}The work on this issue is temporarily at github{color} (lots of experiments
and tweaking):
> Or directly: git clone git://
> {color}
> ----
> RandomizedRunner is a JUnit runner, so it is capable of running @Test-annotated test
cases. It
> respects regular lifecycle hooks such as @Before, @After, @BeforeClass or @AfterClass,
but it
> also adds the following:
> Randomized, but repeatable execution and infrastructure for dealing with randomness:
> - uses pseudo-randomness (so that a given run can be repeated if given the same starting
>   for many things called "random" below,
> - randomly shuffles test methods to ensure they don't depend on each other,
> - randomly shuffles hooks (within a given class) to ensure they don't depend on each
> - base class RandomizedTest provides a number of methods for generating random numbers,
>   and picking random objects from collections (again, this is fully repeatable given
>   initial seed if there are no race conditions),
> - the runner provides infrastructure to augment stack traces with information about the
>   seeds used for running the test, so that it can be repeated (or it can be determined
>   the test is not repeatable -- this indicates a problem with the test case itself).
> Thread control:
> - any threads created as part of a test case are assigned the same initial random seed

>   (repeatability),
> - tracks and attempts to terminate any Threads that are created and not terminated inside

>   a test case (not cleaning up causes a test failure),
> - tracks and attempts to terminate test cases that run for too long (default timeout:
60 seconds,
>   adjustable using global property or annotations),
> Improved validation and lifecycle support:
> - RandomizedRunner uses relaxed contracts of hook methods' accessibility (hook methods
_can_ be
>   private). This helps in avoiding problems with method shadowing (static hooks) or overrides
>   that require tedious super.() chaining). Private hooks are always executed and don't
>   subclasses in any way, period.
> - @Listeners annotation on a test class allows it to hook into the execution progress
and listen
>   to events.
> - @Validators annotation allows a test class to provide custom validation strategies

>   (project-specific). For example a base class can request specific test case naming
>   (or reject JUnit3-like methods, for instance).
> - RandomizedRunner does not "chain" or "suppress" exceptions happening during execution
>   a test case (including hooks). All exceptions are reported as soon as they happened
and multiple
>   failure reports can occur. Most environments we know of then display these failures
>   allowing a clearer understanding of what actually happened first.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message