harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject Re: Test framework
Date Tue, 17 Jan 2006 14:41:38 GMT

Tim Ellison wrote:
> Stepan Mishura wrote:
>> A test framework is quite essential part and we should define it on early
>> stage.
>> I think first we should agree on which test we are going to provide and form
>> requirements to them. There are no doubts that we will provide unit tests
>> for classlib. So unit tests are provided by code authors in JUnit format and
>> intended to verify all code functionality. They are placed in the same
>> package as tested code and run on the bootclasspath.
>> Anything to add? Are there any objections?
> You have seen that there *are* objections to putting the test code into
> the implementation packages and running it on the bootclasspath.  I can
> restate the reasons if it helps.

Yes please.  Because my suggestion of a test framework was directly 
targeted at that well-founded objection of yours.

And I do think we need to step back and think about unit tests vs API 
tests vs functional tests vs whatever.

>> Conformance tests we will receive form TCK and what about other test suites:
>> functional, stress tests, performance? Will we create a separate framework
>> for each?
> Running the TCK is a certification step, not a suite of tests that we
> develop against to get a progressively compliant implementation.

That is correct, but we will be using the TCK as part of our CI.  I 
learned that lesson well in Geronimo.

> I believe we need:
> - implementation tests : using the 'trick' that Geir refers to below to
> expose some internal APIs to the application code.  We would only expect
> these to pass on Harmony code.

yes.  This is normally referred to as a 'unit test'.  No trick is needed 
if we have a test harness to run in, rather than putting on the boot 
classpath of the VM du jour.

> - API unit tests : driving the published APIs, these should pass on all
> compliant implementations (modulo known bugs in those impls.).

yes - these are the "API tests", which IMO do *not* belong in the java.* 

> - then, as you say, additional 'system', 'stress', 'performance', tests
> which are likely to fall outside the unit test framework.


I'd prefer if we distinguished between 'unit tests', which do test the 
internals of the implementations, and 'API tests', which live on the API 
surface area.  No internals, no package, no private, etc.


> Regards,
> Tim
>> Thanks,
>> Stepan Mishura,
>> Intel Middleware Products Division
>> On 1/16/06, Geir Magnusson Jr <geir@pobox.com> wrote:
>>> One thing that's popped up on the "Test suite layout" thread is the
>>> thought that we need to b0rk the canonical package and naming
>>> conventions for unit tests in order to be able to run things on the boot
>>> classpath of the VM.  I think this issue is important enough and
>>> fundamental enough to warrant it's own thread.
>>> First, do we really need to do this?  I thought that we (Tim and I) had
>>> informally discussed this at ApacheCon and came to some good conclusion
>>> where we were able to figure out a trick.
>>> Second, shouldn't we think about providing a test environment in which
>>> we can completely control the environment  - we can test the class
>>> library in a container that can be run in any VM so we have full control
>>> over security and other issues?
>>> Of course, I'd like both.  If we do have the "trick" that we talked
>>> about, then we can use canonical JUnit (or TestNG) naming and package
>>> conventions, which I think is important.
>>> geir

View raw message