incubator-kato-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bobrovsky, Konstantin S" <konstantin.s.bobrov...@intel.com>
Subject Re: TCK and all that
Date Fri, 30 Oct 2009 13:31:09 GMT
Stuart,

I agree, there are lot of obstacles with the self-introspection (I like this term you used)
approach to TCK tests.

> The only frustration is with 1.4.2, which doesn't provide support for 
> Thread.getAllStackTraces(), JVMTI, and no doubt
> other things. I think that is something that should be up for discussion.

This can probably be worked around in 1.4.2: either Thread.enumerate + Thread.getThreadGroup
+ ThreadGroup.getParent could be used, or the test application could keep track of all spawned
threads (this way pre-existing threads would be missed, of course, which reduces strength
of the test)

BTW, presence of JVMPI in Hotspot1.4.2 does not help us, as we want something standard supported
by all major 1.4.2 JVMs, right?

> Using JVMTI to store a file with the JVM state is an interesting 
> suggestion. If we take the JVM state and write it out to a file with a 
> JVMTI agent, what we end up producing might end up being the CJVMTI 
> implementation of Kato that we have just now, which does raise the 
> question of how we go about testing that. It is interesting that would 
> could use the JVMTI agent to verify alternative implementations of the 
> API.

This is an interesting idea - for the step (1) of the TCK test to produce "golden" information
dumps in the format understandable by KATO implementation directly!

I meant slightly different thing, though: I was talking about verifying "post-mortem" analysis
capabilities of the tested KATO implementation at the 2nd step of the TCK. I.e. the test would
first generate a core dump (or memory dump on Windows) of the test application, parse it using
the tested KATO implementation and compare the result with the "golden" output generated at
step (1). JVMTI usage here would provide opportunities for stronger tests, as it has access
to more JVM internals than the standard Java API has.

> I won't go into more detail, but I think that introspecting from within
> Java is not going to get us too far.

I agree, not "too far" (i.e. exhaustive tests checking every aspect of KATO API implementation
would hardly be possible), but some good tests covering some part of the API can still benefit
from using this approach, especially taking into account absence of JVMTI in 1.4.2.

Thanks,
Konst

Intel Novosibirsk
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park, 
17 Krylatskaya Str., Bldg 4, Moscow 121614, 
Russian Federation

---------------------------------

Hi Konstantin,
   Thanks for your reply.  The existing TCK works by running the 
configuration
methods on JUnit test cases and then creating a dump. The JUnit test classes
are run again, only within the JUnit framework such that the test 
methods are
executed this time. The advantage of this system is that the set up and 
the checking
both exist within the same class file and so consistency is easily 
maintained. Now,
one of the problems with that approach is that the both the setup and 
the testcases have
to be compiled and run using at least Java 5.0, and yet we wish to 
support Java 1.4.2.
The other problem is that what we look for is what we expect from our 
program's configuration
of the JVM, and no more, so your suggestion for the JVM to introspect 
itself while running
does potentially produce a more complete test of the API's abilities. 
The only frustration
is with 1.4.2, which doesn't provide support for 
Thread.getAllStackTraces(), JVMTI, and no doubt
other things. I think that is something that should be up for discussion.

Using JVMTI to store a file with the JVM state is an interesting 
suggestion. If we take the JVM
state and write it out to a file with a JVMTI agent, what we end up 
producing might end up being
the CJVMTI implementation of Kato that we have just now, which does 
raise the question of how we
go about testing that. It is interesting that would could use the JVMTI 
agent to verify alternative
implementations of the API.

Introspecting within Java could do:
    o Thread.getAllStackTraces()
       as you suggested. Provided the threads we are interested in are 
parked, there should be some
       consistency, although the time between calling 
getAllStackTraces() and producing a dump could
       produce inconsistencies, which might be just the thread calling 
getAllStackTraces().
       Cross-referencing class names in the stack-traces with their 
class instances may be very difficult,
       unless uniqueness can be guaranteed. I think JVMTI would be 
better suited to this.

    o Reflection
       We could retrieve parts of the heap and class hierarchy using 
reflection. However, private fields would be
       problematic.

    o java.lang.ClassLoader
       The problem with this class is that it doesn't tell you what it 
loaded, and it can only
       tell you about the parent loader or the system classloader. So it 
doesn't really allow complete
       knowledge of the system.

There are certain things we can do within Java for the Image API:
    o Image.getHostName(), Image.getIPAddreses()
       could be easily fulfilled with the Java API.
    o Image.getSystemType(),Image.getSystemSubType()
       could be fulfilled using the System properties (e.g. os.name, 
os.arch, os.version)
       In addition, the files on the system could be read to work out 
what system it is.
    o Image.getInstalledMemory()
       With knowledge of the OS, this might be available through shell 
commands or available
       in a special file. (e.g. /proc/meminfo)
    o Image.getProcessorCount(), getProcessorType(), getProcessorSubType()
       Like the previous entry, from Java we could call getconf, or look
       under /proc/cpuinfo on linux. I assume there are other
    o ImageProcess.getEnvironment()
        Can be comfortable implemented using System.getenv()
    o ImageAddressSpace.getSections()
       All I can think of is the /proc/$$/maps file. Otherwise

To introspect the system more thoroughly JNI could be used to query the 
OS directly:
    o ImageProcess.getID()
       A JNI function to call "getpid()" on UNIX or Linux platforms and 
equivalent elsewhere.

The particularly painful points will be:
    o ImageSection
       In general, I don't see how these can be implemented without 
knowledge from the JVM.
    o ImageThread
       Retrieving these and correlating them with JavaThread instances 
is going to be difficult.
       A Java thread executing a JNI thread could correlate a native 
thread and Java thread, but
       all of the other threads will have difficulty.

The difficulties above will all be shared by the CJVMTI implementation, 
of course.

I won't go into more detail, but I think that introspecting from within 
Java is not going to get us too far. Using JVMTI should produce
more useful information, and supplementing that with other functions to 
introspect on the system itself would also be used by the CJVMTI
implementation of the RI. Of course, we might have two codebases that do 
much the same thing to check against one another, so I think that
having explicit tests like we do just now (if only they were working...) 
would still be prudent to eliminate the possibility of consistent errors and
ensure compliance with the specification.



Regards,
    Stuart



Bobrovsky, Konstantin S wrote:
> Hi Stuart,
>
> one of the approaches to write TCK tests for the "1.0" mode I can think of
> is making the test application retrieve an log information to be checked
> (using standard Java API means) to some "golden" file, then expecting the
> TCK test obtain the same (or "similar") information from a "snapshot" using
> the RI API.
>
> In more details:
> There are several agents:
>  - KATO implementation to be tested
>  - Java Runtime which runs the tests and is coupled with the KATO
>    implementation
>  - test application (a set of classes run as a sample payload application)
>  - TCK test, which can work in 2 modes:
>    (1) run the test application, retrieve and log all the information
>        to be checked in mode (2) to a "golden file" (should be stable
>        between invokations), have the snapshot to be generated by some
>        means.
>    (2) use RI to read the snapshot and retrieve necessary information, then
>        Compare what's retrieved with the golden file
> Each such test is run 2 times: in mode (1) to generate the golden file and
> the dump, and in mode (2) when actual API implementation verification
> happens.
>
> Of course, not entire API can be tested this way, but a fair part can. The main question
is what kind of information can be obtained in mode (1), which can later be retrieved from
the snapshot. What comes to mind is:
>   - all alive threads running in the test application and their Java stacks
>     using java.lang.Thread.getAllStackTraces. For threads with pre-defined
>     names  which the test recognizes as "special" ones (and which are
>     created by the test application), some field values or other details
>     can be queried
>   - a subset of live objects created by the test application
>   - ... many more if JVMTI or JMX is used as the information provider
>     in mode (1)
>
> Thanks,
> Konst
>  
> Intel Novosibirsk
> Closed Joint Stock Company Intel A/O
> Registered legal address: Krylatsky Hills Business Park, 
> 17 Krylatskaya Str., Bldg 4, Moscow 121614, 
> Russian Federation
>  
>
>   
>> -----Original Message-----
>> From: Stuart Monteith [mailto:stukato@stoo.me.uk]
>> Sent: Thursday, October 22, 2009 9:51 PM
>> To: kato-dev@incubator.apache.org; kato-spec@incubator.apache.org
>> Subject: TCK and all that
>>
>> Hi everybody,
>>
>>    Apologies for the cross posting - but the implementation of the TCK
>> is relevant to both camps.
>> One of the things we need to do before making a release available is to
>> produce a TCK. By "TCK" I mean
>> a test harness that will:
>>
>> 1. Setup a JVM into a known configuration and generate a dump.
>> 2. Open a dump with the Kato API and run unit tests against it.
>>
>> Whether or not these are just functional tests or technology compliance
>> is something I will just gloss over for now
>> as just a classification issue.
>>
>> As we all know, there is a problem with optionality in the API. There is
>> information that will be lost when using
>> certain dump types.
>>
>> Steve has suggested two modes for running the TCK.
>>
>> Legacy
>> ========
>>
>> This mode runs through the API and confirms that the basic API behaves
>> as it is intended to do so.
>> It checks that the rules of the API are adhered to, but doesn't expect
>> any method will return any
>> particular piece of information.
>>
>> For example:
>>    List<ImageProcess> ImageAddressSpace.getProcesses();
>>
>> processes would be checked to ensure that it is not null. If there were
>> contents
>> they would be queried, and their methods executed.
>>
>> For example:
>>    String ImageProcess.getID();
>>
>> Would be expected to return a String or throw a DataUnavailable or
>> CorruptDataException exception.
>> The actual contents of the string wouldn't be checked - just so long as
>> it isn't null.
>> (incidentally, actually checking the value in this example for
>> correctness is difficult).
>>
>> 1.0
>> =====
>>
>> This is a test for exact compliance with the API. All of the information
>> that can be retrieved from the dump
>> must be retrieved and it must all be correct.
>>
>> With some caveats.
>>
>> Like with ImageProcess.getID(), there is some information that is poorly
>> specified or difficult to know in
>> advance.
>>
>> The poorly specified aspects of the API are:
>>    Platform specific items.
>>       For example:
>>
>>    Implementation specific items:
>>       toString() methods - what do they print?
>>
>> Things that are difficult to know in advance:
>>    Native stack frames - ordering and naming.
>>    JavaThreads - there will be threads that are implementation specific.
>>
>> There problems aren't intractable. A certain amount of flexibility will
>> be necessary to accommodate variations.
>> We should expect there to be more than what we expect, so we should just
>> look for what we expect to be there,
>> and ignore the rest.
>>
>> The poorly specified aspects of the API should just be more precisely
>> specified. Although it means that the TCK will
>> have to be aware of the different platforms and their behaviour.
>>
>>
>> I'm not a great fan of the "Legacy" mode. While it may have a basic use for
>> testing basic behaviour, I would like for there to be stronger statement
>> about what we expect the  API to do.
>>
>> For example, if we call JavaRuntime.getThreads(), I would expect a
>> List<JavaThread> to be returned. However, if it wasn't empty, I would want
>> the threads we put into the testcase to be there and identifiable by their
>> (correct) names.
>>
>> i.e. if there is something to test we should test it.
>>
>> I call this mode "Permissive" (alternatives: lenient, acquiescent, lax,
>> liberal, tolerant)
>>
>> There are, of course, problems. But I would expect that is this mode wasn't
>> possible or advisable, it would be improbable that a program could be
>> writtent to call the API.
>>
>> The end result is that if you get information out of the API, it must be
>> correct. Where you don't, it is ignored.
>>
>> Thoughts?
>>
>>
>> Thoughts?
>>
>> Regards,
>> 	Stuart
>>
>>
>>
>> --
>> Stuart Monteith
>> http://blog.stoo.me.uk/
>>     
>
>   

-- 
Stuart Monteith
http://blog.stoo.me.uk/




Mime
View raw message