lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dawid Weiss (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4463) Add support for running the same test method/class many times with different class seeds
Date Sat, 06 Oct 2012 20:15:02 GMT


Dawid Weiss commented on LUCENE-4463:

I thought about it for a bit longer and exercised a few scenarios. The problem is that I designed
everything (and I mean everything) with two ideas in mind:
- every random element (be it a selection of components, shuffling of order or whatever) is
a derivative of a single "master" seed. This seed is picked by <junit4> task and is
then used to sort suites to be executed, pick parameters, then is passed to suites to log
messages, stack traces, etc.
- execution of a test suite (in the sense of a single class) is isolated from anything else
-- any other class running before or after. So you can provide the same master seed for a
single class and it should execute identically, even if it's detached from the entire sequence
of suites than ran during the full test. The "seed decorators" that we currently use alter
the master seed with a hash of the test class's name to make it different for each class running
under the same master seed, but this is an independent operation -- whether something ran
before or after doesn't matter.

The idea of running the same suite many times with a _different_ master seed each time conflicts
with these assumptions because then every subsequent execution of the same class will _not_
be a derivative of the master seed anymore (and will most likely depend on how many classes
executed before or even be random). 

Let me illustrate this on an example. Let's say the master seed is XXX; we use this seed to
pick file.encoding and for this seed it becomes UTF-8. If we now pick a random master seed
(say, YYY) for concrete class and it fails, it'll report YYY back to the console. But if you
ant -Dtests.seed=YYY then the selection of file.encoding will be different because, ehm, it's
not XXX anymore. file.encoding has to be picked before the JVM is started so it cannot be
done from within the running test runner, etc.

This is just one of the problem scenarios, there are more but I hope you get the picture.

A "clean" solution to the problem would be to make a loop inside ant, around the contents
of the test-macro (so that the entire sequence of picking the master seed, picking parameters,
spawning JVMs, etc. is repeated). This isn't really going to make matters much faster because
it'll fork new JVMs etc.

A "dirty" solution is to screw the above idealistic point of view and have a seed decorator
which affects the master seed before it is propagated to each suite. This will cause all the
headaches mentioned above PLUS you'll have to get the failing seed directly from the failing
test (stack trace or whatever other message is printed) because it won't be the master seed
JUnit4 greets you with. Then you could indeed run as many concurrent instances of the same
suite with random seeds as you like (JVMs reused). This does sound like super-advanced and
convoluted piece of functionality for something that will be probably used pretty frequently
(which means lots of wtfs on the mailing list).

Don't know, really.

> Add support for running the same test method/class many times with different class seeds
> ----------------------------------------------------------------------------------------
>                 Key: LUCENE-4463
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Wish
>          Components: general/build
>            Reporter: Robert Muir
>            Assignee: Dawid Weiss
>         Attachments: LUCENE-4463.patch
> I have a shell script for this, mike has a python script, its annoying :)
> I want to do something like this:
> ant beast -Dtestcase=XXXX -Dtestmethod=YYYY -Diterations=100
> I would be happy with a simple loop that just invokes 'test' somehow: getting a fresh
new JVM to each iteration is desirable anyway (so you get fresh codecs, etc). 
> the -Dtests.iters is not really useful for this because it does not allow -Dtestmethod
and it does not give a fresh jvm.
> bonus points if it can use multiple jvms at the same time though :)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message