From kato-dev-return-245-apmail-incubator-kato-dev-archive=incubator.apache.org@incubator.apache.org Thu Oct 29 11:29:48 2009 Return-Path: Delivered-To: apmail-incubator-kato-dev-archive@minotaur.apache.org Received: (qmail 14946 invoked from network); 29 Oct 2009 11:29:48 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 29 Oct 2009 11:29:48 -0000 Received: (qmail 95434 invoked by uid 500); 29 Oct 2009 11:29:48 -0000 Delivered-To: apmail-incubator-kato-dev-archive@incubator.apache.org Received: (qmail 95349 invoked by uid 500); 29 Oct 2009 11:29:47 -0000 Mailing-List: contact kato-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: kato-dev@incubator.apache.org Delivered-To: mailing list kato-dev@incubator.apache.org Received: (qmail 95329 invoked by uid 99); 29 Oct 2009 11:29:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 11:29:47 +0000 X-ASF-Spam-Status: No, hits=-6.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [192.55.52.88] (HELO mga01.intel.com) (192.55.52.88) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 11:29:44 +0000 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP; 29 Oct 2009 04:22:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.44,645,1249282800"; d="scan'208";a="508841652" Received: from irsmsx601.ger.corp.intel.com ([163.33.7.164]) by fmsmga002.fm.intel.com with ESMTP; 29 Oct 2009 04:20:47 -0700 Received: from irsmsx504.ger.corp.intel.com ([163.33.3.240]) by irsmsx601.ger.corp.intel.com ([163.33.7.164]) with mapi; Thu, 29 Oct 2009 11:28:21 +0000 From: "Bobrovsky, Konstantin S" To: "kato-spec@incubator.apache.org" , "kato-dev@incubator.apache.org" Date: Thu, 29 Oct 2009 11:28:13 +0000 Subject: RE: TCK and all that Thread-Topic: TCK and all that Thread-Index: AcpTJ0PPDWYRfA/1R2ydClRd0fYS5QFXH7Ew Message-ID: References: <4AE07161.8080302@stoo.me.uk> In-Reply-To: <4AE07161.8080302@stoo.me.uk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Hi Stuart, one of the approaches to write TCK tests for the "1.0" mode I can think of is making the test application retrieve an log information to be checked (using standard Java API means) to some "golden" file, then expecting the TCK test obtain the same (or "similar") information from a "snapshot" using the RI API. In more details: There are several agents: - KATO implementation to be tested - Java Runtime which runs the tests and is coupled with the KATO implementation - test application (a set of classes run as a sample payload application) - TCK test, which can work in 2 modes: (1) run the test application, retrieve and log all the information to be checked in mode (2) to a "golden file" (should be stable between invokations), have the snapshot to be generated by some means. (2) use RI to read the snapshot and retrieve necessary information, then Compare what's retrieved with the golden file Each such test is run 2 times: in mode (1) to generate the golden file and the dump, and in mode (2) when actual API implementation verification happens. Of course, not entire API can be tested this way, but a fair part can. The = main question is what kind of information can be obtained in mode (1), whic= h can later be retrieved from the snapshot. What comes to mind is: - all alive threads running in the test application and their Java stacks using java.lang.Thread.getAllStackTraces. For threads with pre-defined names which the test recognizes as "special" ones (and which are created by the test application), some field values or other details can be queried - a subset of live objects created by the test application - ... many more if JVMTI or JMX is used as the information provider in mode (1) Thanks, Konst =20 Intel Novosibirsk Closed Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park,=20 17 Krylatskaya Str., Bldg 4, Moscow 121614,=20 Russian Federation =20 >-----Original Message----- >From: Stuart Monteith [mailto:stukato@stoo.me.uk] >Sent: Thursday, October 22, 2009 9:51 PM >To: kato-dev@incubator.apache.org; kato-spec@incubator.apache.org >Subject: TCK and all that > >Hi everybody, > > Apologies for the cross posting - but the implementation of the TCK >is relevant to both camps. >One of the things we need to do before making a release available is to >produce a TCK. By "TCK" I mean >a test harness that will: > >1. Setup a JVM into a known configuration and generate a dump. >2. Open a dump with the Kato API and run unit tests against it. > >Whether or not these are just functional tests or technology compliance >is something I will just gloss over for now >as just a classification issue. > >As we all know, there is a problem with optionality in the API. There is >information that will be lost when using >certain dump types. > >Steve has suggested two modes for running the TCK. > > Legacy >=3D=3D=3D=3D=3D=3D=3D=3D > >This mode runs through the API and confirms that the basic API behaves >as it is intended to do so. >It checks that the rules of the API are adhered to, but doesn't expect >any method will return any >particular piece of information. > >For example: > List ImageAddressSpace.getProcesses(); > >processes would be checked to ensure that it is not null. If there were >contents >they would be queried, and their methods executed. > >For example: > String ImageProcess.getID(); > >Would be expected to return a String or throw a DataUnavailable or >CorruptDataException exception. >The actual contents of the string wouldn't be checked - just so long as >it isn't null. >(incidentally, actually checking the value in this example for >correctness is difficult). > > 1.0 >=3D=3D=3D=3D=3D > >This is a test for exact compliance with the API. All of the information >that can be retrieved from the dump >must be retrieved and it must all be correct. > >With some caveats. > >Like with ImageProcess.getID(), there is some information that is poorly >specified or difficult to know in >advance. > >The poorly specified aspects of the API are: > Platform specific items. > For example: > > Implementation specific items: > toString() methods - what do they print? > >Things that are difficult to know in advance: > Native stack frames - ordering and naming. > JavaThreads - there will be threads that are implementation specific. > >There problems aren't intractable. A certain amount of flexibility will >be necessary to accommodate variations. >We should expect there to be more than what we expect, so we should just >look for what we expect to be there, >and ignore the rest. > >The poorly specified aspects of the API should just be more precisely >specified. Although it means that the TCK will >have to be aware of the different platforms and their behaviour. > > >I'm not a great fan of the "Legacy" mode. While it may have a basic use fo= r >testing basic behaviour, I would like for there to be stronger statement >about what we expect the API to do. > >For example, if we call JavaRuntime.getThreads(), I would expect a >List to be returned. However, if it wasn't empty, I would want >the threads we put into the testcase to be there and identifiable by their >(correct) names. > >i.e. if there is something to test we should test it. > >I call this mode "Permissive" (alternatives: lenient, acquiescent, lax, >liberal, tolerant) > >There are, of course, problems. But I would expect that is this mode wasn'= t >possible or advisable, it would be improbable that a program could be >writtent to call the API. > >The end result is that if you get information out of the API, it must be >correct. Where you don't, it is ignored. > >Thoughts? > > >Thoughts? > >Regards, > Stuart > > > >-- >Stuart Monteith >http://blog.stoo.me.uk/