Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: pass (asf.osuosl.org: local policy)
Date: Tue, 24 Jan 2006 21:16:48 +0100
From: Kristian Waagan <Kristian.Waagan@Sun.COM>
Subject: Re: Features of the JUnit test execution harness
In-reply-to: <43D679D1.5040707@sun.com>
To: derby-dev@db.apache.org
Message-id: <43D68B30.5060908@Sun.com>
Organization: Sun Microsystems Inc.
MIME-version: 1.0
Content-type: text/plain; charset=ISO-8859-1; format=flowed
Content-transfer-encoding: 7BIT
User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.7.12) Gecko/20050930
 Thunderbird/1.0.7 Mnenhy/0.7.2.0
References: <43D634B8.7060706@Sun.COM> <43D679D1.5040707@sun.com>

David W. Van Couvering wrote:
> Hi, Kristian, thanks for your questions.  My one overriding thought is 
> we should take this in incremental steps -- do all of these questions 
> need to be answered before we can rewrite a single old canon-based test 
> to a JUnit test?  Can some of these questions be deferred?

Yes, they can be deferred. I did not intend to say that we should try to 
fix/implement all issues/features at in one step.

My main point of concern, is that it is too hard to write JUnit tests 
now, because there is so little information available. Just have a look 
at the number of JUnit tests that have been added to the repository - it 
sure ain't many! The conversion process, which I understand is fully 
based on "it's my itch" initiatives, is also moving along very, very slowly.

> 
> Kristian Waagan wrote:
> 
>> Hello,
>>
>> I have wanted to rewrite a old canon based test to a JUnit test for a 
>> while, but each time I get started I always run into issues that we do 
>> not have guidelines for yet. So, I have a few questions I would like 
>> some feedback on.
>> I know there are a lot of issues to consider, far more then the one I 
>> mention here, but I hope to get a feeling of the general direction we 
>> want to move in.
>>
>>
>> 1) Should we support specific ordering of tests (within a single class)?
>>
>> This feature is wanted if a number of tests operate on shared data and 
>> each test method can be seen as a step in the test. If the order is 
>> mixed up, the intermediate results will not be as expected and all the 
>> tests may fail.
>> In general, assuming anything about the execution order of tests is 
>> strongly disencouraged. However, since we are testing a database 
>> system, we might want to allow for this. This feature can influence 
>> the both the size of the test methods and the execution time for the 
>> test/suite.
> 
> 
> I thought if you name your tests correctly (e.g. test_010, test_020, 
> etc.) then they are run in that order.  Or is that not the case because 
> all method names on a class are hashed?

This is not the case. Tests are not ordered by method names. I do not 
know the implementation details about ordering when using reflection. I 
do know that when adding one test at a time in suite(), they are stored 
in a Vector and thus ordered (and executed in order). But please note 
that this is not a documented feature of JUnit.

> 
> Assuming that's not the case, it seems to me that if you have specific 
> ordering requirements, you could use multiple test classes, create a 
> suite of those classes, and use the TestSetup mechanism you describe 
> below to do any setup work prior to running the suite.
> 

Might work. In my opinion, the best thing is to write each test to be 
totally independent, but I am looking for ways to handle tests that are 
dependent on each other because I suspect that need will arise.

>>
>> 2) Should we support setting up a shared test fixture for a set of tests?
>>
>> This is a common issue with JUnit, and there are mechanisms to handle 
>> it. For instance, we could let tests that require this to wrap itself 
>> in a TestSetup instance (as described in the JUNit FAQ). Letting 
>> DerbyJUnitTest extend TestSetup is another solution, but it might have 
>> unwanted side-effects.
> 
> 
> I think there is a lot of value to having something like this (see above 
> for an example).
> 
>>
>> 3) Many tests just want a database to work on. Other tests want to 
>> create their own database due to specific requirements. How do we make 
>> this easy for the developer?
>>
> 
> I think if you have a TestSetup mechanism, then a suite of tests can 
> depend on the fact that a database has been created.  Creating a 
> database is pretty darn easy already; are there particular things that 
> are repetetive or error-prone?

No. As long as you have the driver name and the connection string 
(including the correct protocol and database path/name), it is very 
easy. This information must be passed on to the test setup class when 
suite() is run, and with the current harness implementation (with 
respect to JUnit tests) this information is not available then. There 
are easy workarounds, but they are not written down anywhere...
This is actually a matter of question 6.

> 
>> 4) Should it be possible to run single JUnit test/suite from the 
>> command line by simply invoking the main method?
>>
>> Personally, I think this should be possible. This require us to remove 
>> the current use of main for running a test in the old test harness, 
>> which I believe is doable with a few changes (in the old harness).
>>
> 
> I think this is essential for debugging and incremental testing.  +1 on 
> this idea.
> 
>> 5) Is the ultimate goal to end up with a pure JUnit based test harness?
>>
>> If this is the goal, we should design for this and accept some extra 
>> work for plugging JUnit tests into the old test harness.
> 
> 
> I can see us getting there over time, but this is a Very Big Effort. Are 
> there particular things we have to decide upon now in order for this to 
> be possible in the future?

I don't know of any, but someone else might.
As long as we code the tests in typical JUnit spirit, we should be good. 
As soon as we start deviating from "standard" behavior, people won't be 
able to run the tests in third-party tools (for instance NetBeans).

We might not be able to allow for this anyway, though.

> 
>>
>> 6) Where, and how, do we want to store settings for a test?
>>
> 
> In a database! :)
> 
> I personally think a properties file is easier to work with than an XML 
> file, *as long as* we don't load these in as System properties, gads, 
> that's a nightmare.
> 
>> 7) How is it determined which framework(s) a JUnit test should be run 
>> in? (i.e. embedded, client, JCC)
>>
> 
> I think you should have ways to run a particular suite under a 
> particular framework, as you do today.

I agree. You could also have a way to run under all available 
frameworks, but then a bit more is required by the harness (or manual 
work - starting servers).

> 
>>
>> If you have other opinions or comments related to JUnit testing, 
>> please put them forth :)
>>
> 
> I think ther are two separate questions:
> 
> - Running tests under JUnit with a JUnit harness
> - Rewriting existing tests to no longer use canons
> 
> I think we need to make sure we discuss these separately.  Existing 
> tests can run under JUnit with canons just fine (that's what we're 
> already doing).  I think that's could be big performance improvement 
> right there (I remember Dan showing the difference in performance).
> 

Hmm, I thought we were running JUnit tests under the old harness. That 
is, the harness automatically generates an empty canon.

Anyway, I agree that if our current harness gets the job done, there is 
no need to rush for a new harness capable of running only JUnit tests. 
But I mean we need to produce enough information to enable people to 
write JUnit tests quickly and without great suffering ;)


--
Kristian

> Rewriting tests to no longer use canons (and thus .sql files) is a Much 
> Bigger Task and I suggest that be separated out, and we should discuss 
> its merits/demerits separately.
> 
>> I hope we are able to get a discussion on all or some of these issues, 
>> and that it results in a Wiki page describing JUnit testing for Derby. 
>> I created a page here: http://wiki.apache.org/db-derby/DerbyJUnitTesting
>>
> 
> I am sure you will get a discussion.  In terms of it "resulting" in a 
> Wiki page, I think that is up to whoever has the itch to do that (it 
> sounds like you already do).
> 
> All the best,
> 
> David
> 
>>
>>
>> -- 
>> Kristian