incubator-kato-spec mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bobrovsky, Konstantin S" <konstantin.s.bobrov...@intel.com>
Subject RE: TCK and all that
Date Thu, 29 Oct 2009 11:31:47 GMT
>   (2) use RI to read the snapshot and retrieve necessary information, then

Not 'RI', but 'KATO implementation to be tested', of course. Sorry for confusion.

Thanks,
Konst
 
Intel Novosibirsk
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park, 
17 Krylatskaya Str., Bldg 4, Moscow 121614, 
Russian Federation
 

>-----Original Message-----
>From: Bobrovsky, Konstantin S [mailto:konstantin.s.bobrovsky@intel.com]
>Sent: Thursday, October 29, 2009 5:28 PM
>To: kato-spec@incubator.apache.org; kato-dev@incubator.apache.org
>Subject: RE: TCK and all that
>
>Hi Stuart,
>
>one of the approaches to write TCK tests for the "1.0" mode I can think of
>is making the test application retrieve an log information to be checked
>(using standard Java API means) to some "golden" file, then expecting the
>TCK test obtain the same (or "similar") information from a "snapshot" using
>the RI API.
>
>In more details:
>There are several agents:
> - KATO implementation to be tested
> - Java Runtime which runs the tests and is coupled with the KATO
>   implementation
> - test application (a set of classes run as a sample payload application)
> - TCK test, which can work in 2 modes:
>   (1) run the test application, retrieve and log all the information
>       to be checked in mode (2) to a "golden file" (should be stable
>       between invokations), have the snapshot to be generated by some
>       means.
>   (2) use RI to read the snapshot and retrieve necessary information, then
>       Compare what's retrieved with the golden file
>Each such test is run 2 times: in mode (1) to generate the golden file and
>the dump, and in mode (2) when actual API implementation verification
>happens.
>
>Of course, not entire API can be tested this way, but a fair part can. The
>main question is what kind of information can be obtained in mode (1),
>which can later be retrieved from the snapshot. What comes to mind is:
>  - all alive threads running in the test application and their Java stacks
>    using java.lang.Thread.getAllStackTraces. For threads with pre-defined
>    names  which the test recognizes as "special" ones (and which are
>    created by the test application), some field values or other details
>    can be queried
>  - a subset of live objects created by the test application
>  - ... many more if JVMTI or JMX is used as the information provider
>    in mode (1)
>
>Thanks,
>Konst
>
>Intel Novosibirsk
>Closed Joint Stock Company Intel A/O
>Registered legal address: Krylatsky Hills Business Park,
>17 Krylatskaya Str., Bldg 4, Moscow 121614,
>Russian Federation
>
>
>>-----Original Message-----
>>From: Stuart Monteith [mailto:stukato@stoo.me.uk]
>>Sent: Thursday, October 22, 2009 9:51 PM
>>To: kato-dev@incubator.apache.org; kato-spec@incubator.apache.org
>>Subject: TCK and all that
>>
>>Hi everybody,
>>
>>    Apologies for the cross posting - but the implementation of the TCK
>>is relevant to both camps.
>>One of the things we need to do before making a release available is to
>>produce a TCK. By "TCK" I mean
>>a test harness that will:
>>
>>1. Setup a JVM into a known configuration and generate a dump.
>>2. Open a dump with the Kato API and run unit tests against it.
>>
>>Whether or not these are just functional tests or technology compliance
>>is something I will just gloss over for now
>>as just a classification issue.
>>
>>As we all know, there is a problem with optionality in the API. There is
>>information that will be lost when using
>>certain dump types.
>>
>>Steve has suggested two modes for running the TCK.
>>
>> Legacy
>>========
>>
>>This mode runs through the API and confirms that the basic API behaves
>>as it is intended to do so.
>>It checks that the rules of the API are adhered to, but doesn't expect
>>any method will return any
>>particular piece of information.
>>
>>For example:
>>    List<ImageProcess> ImageAddressSpace.getProcesses();
>>
>>processes would be checked to ensure that it is not null. If there were
>>contents
>>they would be queried, and their methods executed.
>>
>>For example:
>>    String ImageProcess.getID();
>>
>>Would be expected to return a String or throw a DataUnavailable or
>>CorruptDataException exception.
>>The actual contents of the string wouldn't be checked - just so long as
>>it isn't null.
>>(incidentally, actually checking the value in this example for
>>correctness is difficult).
>>
>> 1.0
>>=====
>>
>>This is a test for exact compliance with the API. All of the information
>>that can be retrieved from the dump
>>must be retrieved and it must all be correct.
>>
>>With some caveats.
>>
>>Like with ImageProcess.getID(), there is some information that is poorly
>>specified or difficult to know in
>>advance.
>>
>>The poorly specified aspects of the API are:
>>    Platform specific items.
>>       For example:
>>
>>    Implementation specific items:
>>       toString() methods - what do they print?
>>
>>Things that are difficult to know in advance:
>>    Native stack frames - ordering and naming.
>>    JavaThreads - there will be threads that are implementation specific.
>>
>>There problems aren't intractable. A certain amount of flexibility will
>>be necessary to accommodate variations.
>>We should expect there to be more than what we expect, so we should just
>>look for what we expect to be there,
>>and ignore the rest.
>>
>>The poorly specified aspects of the API should just be more precisely
>>specified. Although it means that the TCK will
>>have to be aware of the different platforms and their behaviour.
>>
>>
>>I'm not a great fan of the "Legacy" mode. While it may have a basic use
>for
>>testing basic behaviour, I would like for there to be stronger statement
>>about what we expect the  API to do.
>>
>>For example, if we call JavaRuntime.getThreads(), I would expect a
>>List<JavaThread> to be returned. However, if it wasn't empty, I would want
>>the threads we put into the testcase to be there and identifiable by their
>>(correct) names.
>>
>>i.e. if there is something to test we should test it.
>>
>>I call this mode "Permissive" (alternatives: lenient, acquiescent, lax,
>>liberal, tolerant)
>>
>>There are, of course, problems. But I would expect that is this mode
>wasn't
>>possible or advisable, it would be improbable that a program could be
>>writtent to call the API.
>>
>>The end result is that if you get information out of the API, it must be
>>correct. Where you don't, it is ignored.
>>
>>Thoughts?
>>
>>
>>Thoughts?
>>
>>Regards,
>>	Stuart
>>
>>
>>
>>--
>>Stuart Monteith
>>http://blog.stoo.me.uk/


Mime
View raw message