lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dawid Weiss (Resolved) (JIRA)" <>
Subject [jira] [Resolved] (LUCENE-3492) Extract a generic framework for running randomized tests.
Date Tue, 14 Feb 2012 15:33:59 GMT


Dawid Weiss resolved LUCENE-3492.

       Resolution: Fixed
    Fix Version/s:     (was: 4.0)

I consider this issue done. I've pulled all the super-cool features from Lucene/Solr and put
them into a separate, stand-alone and reusable project called randomizedtesting. We have switched
our infrastructure at Carrot2 to use it and I'm very happy with it.

I will file another issue concerned with progressively moving from LuceneTestRunner/Case and
tests infrastructure to RandomizedRunner (and siblings).
> Extract a generic framework for running randomized tests.
> ---------------------------------------------------------
>                 Key: LUCENE-3492
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Minor
>         Attachments: Screen Shot 2011-10-06 at 12.58.02 PM.png
> {color:red}The work on this issue is temporarily at github{color} (lots of experiments
and tweaking):
> Or directly: git clone git://
> {color}
> ----
> RandomizedRunner is a JUnit runner, so it is capable of running @Test-annotated test
cases. It
> respects regular lifecycle hooks such as @Before, @After, @BeforeClass or @AfterClass,
but it
> also adds the following:
> Randomized, but repeatable execution and infrastructure for dealing with randomness:
> - uses pseudo-randomness (so that a given run can be repeated if given the same starting
>   for many things called "random" below,
> - randomly shuffles test methods to ensure they don't depend on each other,
> - randomly shuffles hooks (within a given class) to ensure they don't depend on each
> - base class RandomizedTest provides a number of methods for generating random numbers,
>   and picking random objects from collections (again, this is fully repeatable given
>   initial seed if there are no race conditions),
> - the runner provides infrastructure to augment stack traces with information about the
>   seeds used for running the test, so that it can be repeated (or it can be determined
>   the test is not repeatable -- this indicates a problem with the test case itself).
> Thread control:
> - any threads created as part of a test case are assigned the same initial random seed

>   (repeatability),
> - tracks and attempts to terminate any Threads that are created and not terminated inside

>   a test case (not cleaning up causes a test failure),
> - tracks and attempts to terminate test cases that run for too long (default timeout:
60 seconds,
>   adjustable using global property or annotations),
> Improved validation and lifecycle support:
> - RandomizedRunner uses relaxed contracts of hook methods' accessibility (hook methods
_can_ be
>   private). This helps in avoiding problems with method shadowing (static hooks) or overrides
>   that require tedious super.() chaining). Private hooks are always executed and don't
>   subclasses in any way, period.
> - @Listeners annotation on a test class allows it to hook into the execution progress
and listen
>   to events.
> - @Validators annotation allows a test class to provide custom validation strategies

>   (project-specific). For example a base class can request specific test case naming
>   (or reject JUnit3-like methods, for instance).
> - RandomizedRunner does not "chain" or "suppress" exceptions happening during execution
>   a test case (including hooks). All exceptions are reported as soon as they happened
and multiple
>   failure reports can occur. Most environments we know of then display these failures
>   allowing a clearer understanding of what actually happened first.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message