lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dawid Weiss (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3785) Replace ant macros for running concurrent tests with ant-junit4.
Date Wed, 15 Feb 2012 21:56:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208826#comment-13208826
] 

Dawid Weiss commented on LUCENE-3785:
-------------------------------------

An example improvement over trunk. I've designed a "super-failing" set of classes that fail
at various points (initializations, rules, hooks). The output from the current trunk is interleaved
and hard to read (compare trunk1.txt and trunk2.txt -- they're executions with the same seed;
it's hard to tell which line of output is associated with which test). The output from junit4
is also reordered (because parallel tests execution is effectively a race condition...), but
the OUTPUT passed to listeners (in this case the console output) is always fully consistent.


Compare junit4-1.txt and junit4-2.txt. Every suite block is atomic. Like this one:
{noformat}
   [junit4] Running org.apache.lucene.util.junitcompat.failures.TestFailOn07After
   [junit4] ERROR   0.09s S3 | TestFailOn07After.testMethod
   [junit4]    > Throwable #1: java.lang.RuntimeException: Failure on @After.
   [junit4]    > 	at org.apache.lucene.util.junitcompat.failures.TestFailOn07After.after(TestFailOn07After.java:13)
   [junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:601)
   [junit4]    > 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
   [junit4]    > 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   [junit4]    > 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
   [junit4]    > 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase$SubclassSetupTeardownRule$1.evaluate(LuceneTestCase.java:700)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase$InternalSetupTeardownRule$1.evaluate(LuceneTestCase.java:599)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase$TestResultInterceptorRule$1.evaluate(LuceneTestCase.java:504)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase$RememberThreadRule$1.evaluate(LuceneTestCase.java:562)
   [junit4]    > 	at org.junit.rules.RunRules.evaluate(RunRules.java:18)
   [junit4]    > 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
   [junit4]    > 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:165)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:57)
   [junit4]    > 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   [junit4]    > 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   [junit4]    > 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   [junit4]    > 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   [junit4]    > 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   [junit4]    > 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
   [junit4]    > 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
   [junit4]    > 	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   [junit4]    > 	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
   [junit4]    > 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:124)
   [junit4]    > 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:186)
   [junit4]    > 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
   [junit4]    > 
   [junit4]   2> NOTE: reproduce with: ant test -Dtestcase=TestFailOn07After -Dtestmethod=testMethod
-Dtests.seed=-737472ec6455983d:77a4c0da1ce834dd:1a2ea2ed4eba1655 -Dargs="-Dfile.encoding=Cp1252"
   [junit4]   2>
   [junit4] Tests run:   1, Failures:   0, Errors:   1, Skipped:   0, Time:  0.25s <<<
FAILURES!
{noformat}

A block like this is composed of sections. It begins with the suite name, followed by tests:
{noformat}
   [junit4] Running org.apache.lucene.util.junitcompat.failures.TestFailOn07After
{noformat}
If anything fails, it is reported as a consistent block within a suite. For example here,
an @After hook failed on testMethod:
{noformat}
   [junit4] ERROR   0.09s S3 | TestFailOn07After.testMethod
   [junit4]    > Throwable #1: java.lang.RuntimeException: Failure on @After.
   [junit4]    > 	at org.apache.lucene.util.junitcompat.failures.TestFailOn07After.after(TestFailOn07After.java:13)
   [junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
   [junit4]    > 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
   [junit4]    > 
   [junit4]   2> NOTE: reproduce with: ant test -Dtestcase=TestFailOn07After -Dtestmethod=testMethod
-Dtests.seed=-737472ec6455983d:77a4c0da1ce834dd:1a2ea2ed4eba1655 -Dargs="-Dfile.encoding=Cp1252"
   [junit4]   2>
{noformat}
An output section like this also follows a pattern. The first line contains a test case's
status, time, slave JVM identifier (S3) and cause description:
{noformat}
   [junit4] ERROR   0.09s S3 | TestFailOn07After.testMethod
{noformat}
then follows (">"-indented) throwable cause and stack:
{noformat}
   [junit4]    > Throwable #1: java.lang.RuntimeException: Failure on @After.
   [junit4]    > 	at org.apache.lucene.util.junitcompat.failures.TestFailOn07After.after(TestFailOn07After.java:13)
...
{noformat}
finally, standards output and standard error streams come at the end. They are also indented
and prefixed with "1>" (stdout" or "2>" (stderr). stderr and stdout calls come in the
order they actually happened on the slave (they're synchronized). In this case, only stderr
was used:
{noformat}
   [junit4]   2> NOTE: reproduce with: ant test -Dtestcase=TestFailOn07After -Dtestmethod=testMethod
-Dtests.seed=-737472ec6455983d:77a4c0da1ce834dd:1a2ea2ed4eba1655 -Dargs="-Dfile.encoding=Cp1252"
   [junit4]   2>
{noformat}

Note that even when when LuceneTestCase fails to provide reproduce string (because the exception
happened outside of @Rule boundaries), the output exception cause is still nicely formatted
and provided. As in this suite class, which even fails to load properly:
{noformat}
   [junit4] Running org.apache.lucene.util.junitcompat.failures.TestFailOn01ClassLoad
   [junit4] ERROR   0.00s S0 | TestFailOn01ClassLoad (suite)
   [junit4]    > Throwable #1: java.lang.ExceptionInInitializerError
   [junit4]    > 	at java.lang.Class.forName0(Native Method)
   [junit4]    > 	at java.lang.Class.forName(Class.java:186)
   [junit4]    > 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:138)
   [junit4]    > 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:116)
   [junit4]    > 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:186)
   [junit4]    > 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
   [junit4]    > Caused by: java.lang.ArithmeticException: / by zero
   [junit4]    > 	at org.apache.lucene.util.junitcompat.failures.TestFailOn01ClassLoad.<clinit>(TestFailOn01ClassLoad.java:11)
   [junit4]    > 	... 6 more
   [junit4]    >
   [junit4] Tests run:   0, Failures:   0, Errors:   1, Skipped:   0, Time:  0.00s <<<
FAILURES!
{noformat}
The summary line contains "Tests run: 0" which may seem incorrect, but in fact no test even
began -- the error comes from the suite loader (the status line contains no method name, just
the suite name).
                
> Replace ant macros for running concurrent tests with ant-junit4.
> ----------------------------------------------------------------
>
>                 Key: LUCENE-3785
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3785
>             Project: Lucene - Java
>          Issue Type: Sub-task
>          Components: general/build, general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Minor
>             Fix For: 4.0
>
>         Attachments: failure-cases.patch, junit4-1.txt, junit4-2.txt, trunk1.txt, trunk2.txt
>
>
> ant-junit4 is an ANT task for running tests in parallel (on slave JVMs). Its advantages
over the current macros:
> - dynamic load balancing based on historical test runs and current execution (job-stealing),
> - jvm-crash resilience (I wrote tests that actually crash a slave jvms to make sure such
an event is reported appropriately),
> - nicer console reporting (no need for syserrs on LuceneTestCase level).
> More documentation and info will follow as I roll out a patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message