lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dawid Weiss (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (LUCENE-3492) Extract a generic framework for running randomized tests.
Date Thu, 06 Oct 2011 17:05:29 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122068#comment-13122068
] 

Dawid Weiss edited comment on LUCENE-3492 at 10/6/11 5:04 PM:
--------------------------------------------------------------

Hi Shai. This is definitely not only for debugging. For example we use randomized testing
inside CarrotSearch to test algorithmic/ combinatorial code. Once you hit a bug, you simply
copy the test case (or a call to a common test case method) and fix the seed to have a regression
test for the future (so that you know you're not failing examples that previously failed).
So, for example:
{code}
@Test @Seed("23095324")
public void runFixedRegression_1 { doSomethingWithRandoms(); }

@Test @Seed("239735923")
public void runFixedRegression_2 { doSomethingWithRandoms(); }

@Test
public void runRandomized { doSomethingWithRandoms(); }
{code}

This is a scenario I really came to like. It's a bit like your tests write themselves for
you :)

I left system properties for fixing seeds and enforcing repetition number because they are
currently in Lucene, although I personally don't like them that much (because they affect
everything globally). I do understand they're useful for quick hacking without recompiling
stuff or for remote executions, but I'd much rather have something like -Dseed.testClass[.method]=xxxx
which would affect only a single class or method rather than everything. The same can be done
for filtering which method/ test case to execute. This is debatable of course and a matter
of personal taste.

I should publish what I have tonight on github (I'm moving certain things out of our proprietary
codebase and there are JUnit corner cases that slow things down).
                
      was (Author: dweiss):
    Hi Shai. This is definitely not only for debugging. For example we use randomized testing
inside CarrotSearch to test algorithmic/ combinatorial code. Once you hit a bug, you simply
copy the test case (or a call to a common test case method) and fix the seed to have a regression
test for the future (so that you know you're not failing examples that previously failed).
So, for example:
{code}
@Test @Seed("23095324")
public void runFixedRegression_1 { doSomethingWithRandoms(); }

@Test @Seed("239735923")
public void runFixedRegression_1 { doSomethingWithRandoms(); }

@Test
public void runRandomized { doSomethingWithRandoms(); }
{code}

This is a scenario I really came to like. It's a bit like your tests write themselves for
you :)

I left system properties for fixing seeds and enforcing repetition number because they are
currently in Lucene, although I personally don't like them that much (because they affect
everything globally). I do understand they're useful for quick hacking without recompiling
stuff or for remote executions, but I'd much rather have something like -Dseed.testClass[.method]=xxxx
which would affect only a single class or method rather than everything. The same can be done
for filtering which method/ test case to execute. This is debatable of course and a matter
of personal taste.

I should publish what I have tonight on github (I'm moving certain things out of our proprietary
codebase and there are JUnit corner cases that slow things down).
                  
> Extract a generic framework for running randomized tests.
> ---------------------------------------------------------
>
>                 Key: LUCENE-3492
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3492
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Minor
>             Fix For: 4.0
>
>         Attachments: Screen Shot 2011-10-06 at 12.58.02 PM.png
>
>
> I love the idea of randomized testing. Everyone (we at CarrotSearch, Lucene and Solr
folks) have their glue to make it possible. The question is if there's something to pull out
that others could share without having the need to import Lucene-specific classes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message