incubator-kato-spec mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuart Monteith <stuk...@stoo.me.uk>
Subject TCK and all that
Date Thu, 22 Oct 2009 14:51:13 GMT
Hi everybody,

    Apologies for the cross posting - but the implementation of the TCK 
is relevant to both camps.
One of the things we need to do before making a release available is to 
produce a TCK. By "TCK" I mean
a test harness that will:

1. Setup a JVM into a known configuration and generate a dump.
2. Open a dump with the Kato API and run unit tests against it.

Whether or not these are just functional tests or technology compliance 
is something I will just gloss over for now
as just a classification issue.

As we all know, there is a problem with optionality in the API. There is 
information that will be lost when using
certain dump types.

Steve has suggested two modes for running the TCK.

 Legacy
========

This mode runs through the API and confirms that the basic API behaves 
as it is intended to do so.
It checks that the rules of the API are adhered to, but doesn't expect 
any method will return any
particular piece of information.

For example:
    List<ImageProcess> ImageAddressSpace.getProcesses();

processes would be checked to ensure that it is not null. If there were 
contents
they would be queried, and their methods executed.

For example:
    String ImageProcess.getID();

Would be expected to return a String or throw a DataUnavailable or 
CorruptDataException exception.
The actual contents of the string wouldn't be checked - just so long as 
it isn't null.
(incidentally, actually checking the value in this example for 
correctness is difficult).

 1.0
=====

This is a test for exact compliance with the API. All of the information 
that can be retrieved from the dump
must be retrieved and it must all be correct.

With some caveats.

Like with ImageProcess.getID(), there is some information that is poorly 
specified or difficult to know in
advance.

The poorly specified aspects of the API are:
    Platform specific items.
       For example:
   
    Implementation specific items:
       toString() methods - what do they print?

Things that are difficult to know in advance:
    Native stack frames - ordering and naming.
    JavaThreads - there will be threads that are implementation specific.

There problems aren't intractable. A certain amount of flexibility will 
be necessary to accommodate variations.
We should expect there to be more than what we expect, so we should just 
look for what we expect to be there,
and ignore the rest.

The poorly specified aspects of the API should just be more precisely 
specified. Although it means that the TCK will
have to be aware of the different platforms and their behaviour.
  

I'm not a great fan of the "Legacy" mode. While it may have a basic use for testing basic
behaviour, I would like for there to be stronger statement about what we expect the  API to
do.

For example, if we call JavaRuntime.getThreads(), I would expect a List<JavaThread>
to be returned. However, if it wasn't empty, I would want the threads we put into the testcase
to be there and identifiable by their (correct) names. 

i.e. if there is something to test we should test it.

I call this mode "Permissive" (alternatives: lenient, acquiescent, lax, liberal, tolerant)

There are, of course, problems. But I would expect that is this mode wasn't possible or advisable,
it would be improbable that a program could be writtent to call the API.

The end result is that if you get information out of the API, it must be correct. Where you
don't, it is ignored.

Thoughts?


Thoughts?

Regards,
	Stuart



-- 
Stuart Monteith
http://blog.stoo.me.uk/


Mime
View raw message