river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Firmstone <j...@zeus.net.au>
Subject Re: JUnit vs. jtreg (Was: Build failed in Hudson: River-trunk #163)
Date Sun, 26 Apr 2009 09:59:58 GMT
Good to have you back Tom,

What I'm afraid of, is an uninformed decision being made, we need to do 
our homework here, so please dive in and check out the code.  Thanks to 
Jonathan asking the right questions, and someone with the knowledge 
replying (thank you Peter Jones for responding) we've now found out 
there are existing Regression and Unit tests that rely on jtreg, in 
addition to the Integration test kit that is now runnable from an ant 
target.

Regression tests are created every time a bug is identified in River's 
API, the conditions under which it occurs are recreated in one or more 
regression tests.  Later when that part of the API is reimplemented, 
(assuming also that we're not deliberately breaking backward 
compatibility) the regression tests confirm that a previously identified 
bug hasn't been reintroduced.  This is important, some of the finer 
implementation details may not be apparent to the future developer, the 
regression test allows us to ensure the bug doesn't get reintroduced at 
a later date.  This is obviously very important for security bugs too.

A problem with reimplementing the regression tests with another tool is 
that we would need the old code that included the bug to confirm that 
the new regression test implementation does in fact reproduce the bug, 
otherwise our efforts would be for naught.

I have a feeling that if we decide to no longer use jtreg for unit tests 
that we may still need to use it for Regression testing, however I'd 
like to reserve any further judgment on my part until my understanding 
improves further.

Tom hobbs wrote:
> Hi guys,
>
> Sorry for my recent silence.
>
> These jtreg doo-hickeys sound like the QA tests to me.  They're 
> something we're inherited and not really sure what to do with them, so 
> I suggest we treat them like the QA tests.
>
> Assuming it's not too difficult, lets get running the jtregs as part 
> of the build.  As River functionality is added/changed and individual 
> jtregs start breaking whoever made the change should assume that it's 
> their responsibility to modify or remove the jtreg (and QA test) as 
> appropriate.
>
> Meanwhile, new code gets junit tested.
>
> Thoughts?
>
> Also, +1 Java 5.  (Java 6 I can take or leave).
>
> Cheers,
>
> Tom
>
>
>
> On Sat, 25 Apr 2009 11:50:10 -0000, Peter Firmstone <jini@zeus.net.au> 
> wrote:
>
>> I hear you.
>>
>> How about we step back, postpone making a decision until we better 
>> understand just what we've inherited then weigh up the pro's and 
>> con's? I haven't seen anyone in a hurry to write junit tests yet, a 
>> little wait while we investigate jtreg can't do any harm? ;) It took 
>> me about 1 hour to download, install and get jtreg running, having 
>> never heard of it before, the tests themselves seem simple, remember 
>> I'm one of your new developers.
>>
>> I'm going to sit on the fence for now have a play and write test 
>> cases for ClassDep using both tools for my own understanding.
>>
>> We're going to have to get to know jtreg to make an informed 
>> decision, here's some info:
>>
>> All tests are placed in the directory tree under qa/jtreg/
>>
>> Ant can be used to run all tests, I'll post the build.xml file after 
>> I write it.
>>
>> The following I've copied from the jtreg website:
>>
>> Ant Examples:
>>
>> <jtreg dir="test/tools/javac"/>
>>
>> Run all the tests in the test/tools/javac directory, using the same 
>> JDK being used to run Ant, the default work and report directories, 
>> and running each test in a separate VM.
>>
>> <jtreg dir="test/tools/javac" workDir="myWorkDir"/>
>>
>> As before, but specifying a work directory for the results.
>>
>> <jtreg dir="test/tools/javac" workDir="myWorkDir">
>>     <include name="api/*.java" />
>> </jtreg>
>>
>> Run just the api/*.java tests.
>>
>> <jtreg>
>>     <arg line="-w myWorkDir -jdk /java/jdk/1.5.0 test/tools/javac">
>> </jtreg>
>>
>> Run the test/tools/javac tests using /java/jdk/1.5.0 writing the 
>> results to myWorkDir.
>>
>> 2. Writing a JDK Regression Test
>> 2.1. How do I write a test?
>>
>> The simplest test is an ordinary Java program with the usual static 
>> main method. If the test fails, it should throw an exception; if it 
>> succeeds, it should return normally.
>>
>> Here's an example:
>>
>>                             /* @test 1.1 97/10/12
>>                                @bug 1234567
>>                                @summary Make sure that 1 != 0
>>                             */
>>
>>                             public class OneZero {
>>
>>                                 public static void main(String[] 
>> args) throws Exception {
>>                                     if (1 == 0) {
>>                                         throw new Exception("1 == 0");
>>                                     }
>>                                 }
>>
>>                             }
>>                        The @test tag identifies a source file that 
>> defines a test. the harness will automatically run any .java, .sh, 
>> and .html file containing an @test tag within the appropriate 
>> comment; it ignores any file not containing such a tag or not 
>> utilizing one of the expected extensions.
>>
>> If necessary the harness will compile the source file, if the class 
>> files are older than the corresponding source files. Other files 
>> which the test depends on must be specified with the @run build action.
>>
>> The arguments to the @test tag are ignored by the harness. For 
>> identification it's useful to put the usual SCCS ID keywords (I and 
>> E, each letter surrounded by %) after the @test tag.
>>
>> While not part of the tag specification, some tests use the string 
>> "/nodynamiccopyright" after @test to indicate that that the file 
>> should not be subject to automated copyright processing that might 
>> affect the operation of the test, for example, by affecting the line 
>> numbers of the test source code.
>> 2.3. What do the other tags mean?
>>
>> The other tags shown above are optional.
>>
>> The @bug tag should be followed by one or more bug numbers, separated 
>> by spaces. The bug number is useful in diagnosing test failures. It's 
>> OK to write tests that don't have bug numbers, but if you're writing 
>> a test for a specific bug please include its number in an @bug tag.
>>
>> The @summary tag describes the condition that is checked by the test. 
>> It is especially useful for non-regression tests, which by definition 
>> don't have bug numbers, but even if there's a bug number it's helpful 
>> to include a summary. Note that a test summary is generally not the 
>> same thing as a Bugtraq synopsis, since the latter describes the bug 
>> rather than the condition that the bug violates.
>>
>>
>>
>> Jeff Ramsdale wrote:
>>> Not to mention tool support--IDEs, etc...
>>> -j
>>>
>>> On Fri, Apr 24, 2009 at 6:13 AM, Jukka Zitting 
>>> <jukka.zitting@gmail.com>wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> On Fri, Apr 24, 2009 at 10:06 AM, Peter Firmstone <jini@zeus.net.au>
>>>> wrote:
>>>>
>>>>> It appears to me that I should perhaps be writing jtreg style unit 
>>>>> tests
>>>>>
>>>> for
>>>>
>>>>> ClassDep rather than JUnit?
>>>>>
>>>> I'm not so sure about that. I can't tell which approach is better
>>>> technically as I don't know jtreg, but from a community building
>>>> perspective it would be much better if we used something that the vast
>>>> majority of the Java world is already familiar with.
>>>>
>>>> BR,
>>>>
>>>> Jukka Zitting
>>>>
>>>>
>>>
>>>
>>
>
>
>


Mime
View raw message