incubator-kato-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bobrovsky, Konstantin S" <konstantin.s.bobrov...@intel.com>
Subject RE: TCK and all that
Date Thu, 29 Oct 2009 11:28:13 GMT
Hi Stuart,

one of the approaches to write TCK tests for the "1.0" mode I can think of
is making the test application retrieve an log information to be checked
(using standard Java API means) to some "golden" file, then expecting the
TCK test obtain the same (or "similar") information from a "snapshot" using
the RI API.

In more details:
There are several agents:
 - KATO implementation to be tested
 - Java Runtime which runs the tests and is coupled with the KATO
   implementation
 - test application (a set of classes run as a sample payload application)
 - TCK test, which can work in 2 modes:
   (1) run the test application, retrieve and log all the information
       to be checked in mode (2) to a "golden file" (should be stable
       between invokations), have the snapshot to be generated by some
       means.
   (2) use RI to read the snapshot and retrieve necessary information, then
       Compare what's retrieved with the golden file
Each such test is run 2 times: in mode (1) to generate the golden file and
the dump, and in mode (2) when actual API implementation verification
happens.

Of course, not entire API can be tested this way, but a fair part can. The main question is
what kind of information can be obtained in mode (1), which can later be retrieved from the
snapshot. What comes to mind is:
  - all alive threads running in the test application and their Java stacks
    using java.lang.Thread.getAllStackTraces. For threads with pre-defined
    names  which the test recognizes as "special" ones (and which are
    created by the test application), some field values or other details
    can be queried
  - a subset of live objects created by the test application
  - ... many more if JVMTI or JMX is used as the information provider
    in mode (1)

Thanks,
Konst
 
Intel Novosibirsk
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park, 
17 Krylatskaya Str., Bldg 4, Moscow 121614, 
Russian Federation
 

>-----Original Message-----
>From: Stuart Monteith [mailto:stukato@stoo.me.uk]
>Sent: Thursday, October 22, 2009 9:51 PM
>To: kato-dev@incubator.apache.org; kato-spec@incubator.apache.org
>Subject: TCK and all that
>
>Hi everybody,
>
>    Apologies for the cross posting - but the implementation of the TCK
>is relevant to both camps.
>One of the things we need to do before making a release available is to
>produce a TCK. By "TCK" I mean
>a test harness that will:
>
>1. Setup a JVM into a known configuration and generate a dump.
>2. Open a dump with the Kato API and run unit tests against it.
>
>Whether or not these are just functional tests or technology compliance
>is something I will just gloss over for now
>as just a classification issue.
>
>As we all know, there is a problem with optionality in the API. There is
>information that will be lost when using
>certain dump types.
>
>Steve has suggested two modes for running the TCK.
>
> Legacy
>========
>
>This mode runs through the API and confirms that the basic API behaves
>as it is intended to do so.
>It checks that the rules of the API are adhered to, but doesn't expect
>any method will return any
>particular piece of information.
>
>For example:
>    List<ImageProcess> ImageAddressSpace.getProcesses();
>
>processes would be checked to ensure that it is not null. If there were
>contents
>they would be queried, and their methods executed.
>
>For example:
>    String ImageProcess.getID();
>
>Would be expected to return a String or throw a DataUnavailable or
>CorruptDataException exception.
>The actual contents of the string wouldn't be checked - just so long as
>it isn't null.
>(incidentally, actually checking the value in this example for
>correctness is difficult).
>
> 1.0
>=====
>
>This is a test for exact compliance with the API. All of the information
>that can be retrieved from the dump
>must be retrieved and it must all be correct.
>
>With some caveats.
>
>Like with ImageProcess.getID(), there is some information that is poorly
>specified or difficult to know in
>advance.
>
>The poorly specified aspects of the API are:
>    Platform specific items.
>       For example:
>
>    Implementation specific items:
>       toString() methods - what do they print?
>
>Things that are difficult to know in advance:
>    Native stack frames - ordering and naming.
>    JavaThreads - there will be threads that are implementation specific.
>
>There problems aren't intractable. A certain amount of flexibility will
>be necessary to accommodate variations.
>We should expect there to be more than what we expect, so we should just
>look for what we expect to be there,
>and ignore the rest.
>
>The poorly specified aspects of the API should just be more precisely
>specified. Although it means that the TCK will
>have to be aware of the different platforms and their behaviour.
>
>
>I'm not a great fan of the "Legacy" mode. While it may have a basic use for
>testing basic behaviour, I would like for there to be stronger statement
>about what we expect the  API to do.
>
>For example, if we call JavaRuntime.getThreads(), I would expect a
>List<JavaThread> to be returned. However, if it wasn't empty, I would want
>the threads we put into the testcase to be there and identifiable by their
>(correct) names.
>
>i.e. if there is something to test we should test it.
>
>I call this mode "Permissive" (alternatives: lenient, acquiescent, lax,
>liberal, tolerant)
>
>There are, of course, problems. But I would expect that is this mode wasn't
>possible or advisable, it would be improbable that a program could be
>writtent to call the API.
>
>The end result is that if you get information out of the API, it must be
>correct. Where you don't, it is ignored.
>
>Thoughts?
>
>
>Thoughts?
>
>Regards,
>	Stuart
>
>
>
>--
>Stuart Monteith
>http://blog.stoo.me.uk/


Mime
View raw message