From kato-spec-return-287-apmail-incubator-kato-spec-archive=incubator.apache.org@incubator.apache.org Tue Dec 01 11:08:43 2009 Return-Path: Delivered-To: apmail-incubator-kato-spec-archive@minotaur.apache.org Received: (qmail 73982 invoked from network); 1 Dec 2009 11:08:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 1 Dec 2009 11:08:43 -0000 Received: (qmail 55630 invoked by uid 500); 1 Dec 2009 11:08:43 -0000 Delivered-To: apmail-incubator-kato-spec-archive@incubator.apache.org Received: (qmail 55589 invoked by uid 500); 1 Dec 2009 11:08:42 -0000 Mailing-List: contact kato-spec-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: kato-spec@incubator.apache.org Delivered-To: mailing list kato-spec@incubator.apache.org Received: (qmail 55579 invoked by uid 99); 1 Dec 2009 11:08:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 11:08:42 +0000 X-ASF-Spam-Status: No, hits=-4.0 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [143.182.124.37] (HELO mga14.intel.com) (143.182.124.37) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 11:08:31 +0000 Received: from azsmga001.ch.intel.com ([10.2.17.19]) by azsmga102.ch.intel.com with ESMTP; 01 Dec 2009 03:08:09 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.47,316,1257148800"; d="scan'208";a="217385958" Received: from irsmsx601.ger.corp.intel.com ([163.33.7.164]) by azsmga001.ch.intel.com with ESMTP; 01 Dec 2009 03:08:08 -0800 Received: from irsmsx504.ger.corp.intel.com ([163.33.3.240]) by irsmsx601.ger.corp.intel.com ([163.33.7.164]) with mapi; Tue, 1 Dec 2009 11:07:58 +0000 From: "Bobrovsky, Konstantin S" To: "kato-spec@incubator.apache.org" Date: Tue, 1 Dec 2009 11:07:57 +0000 Subject: RE: Do we need to change the scope of the API design? Thread-Topic: Do we need to change the scope of the API design? Thread-Index: AcpwRQGLb6jlFnDUTqW1ExbEIbtz9ACI5bHA Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Hi Steve, all, >I realised that when I talk about dropping the Image API its actually >because we can't implement the Java Runtime API by reading from a core >file. Not because we can't read a corefile. Now my sizing was based on >the >need to implement the whole of the API. The killer cost is the >JavaRuntime >piece. You need to be able to understand the internal layout of the JVM >structures and learn how they stick together etc. > >If we agree that there is value in keeping the Image API around as a means >of reading the contents of a core file without being able to get at Java >Runtime information then we could do so. In that vision we'd just remove >the ability to get a JavaRuntime object from the Image API - ie this >method >( >http://hudson.zones.apache.org/hudson/view/Kato/job/kato.api- >head/javadoc/org/apache/kato/image/ImageProcess.html#getRuntimes%28%29) >would go. > >There are other little changes still required but the general idea seems >quite reasonable. - just comes down to the value of corefile readers >themselves > > What do you think? I think this is a good idea. Process-level core file analysis sometimes hel= ps too and it is good to standardize such things as ImageRegister/Process/M= odule/Section, etc. Instead of ImageProcess.getRuntimes, the task of (future) getting a JavaRun= time for a core file by a 3rd party tool can probably be solved in another = way. AFAIU, current RI uses an output from the CJVMTI agent (HPROF alternative) = as the source artifact for building a JavaRuntime based on it - is that cor= rect? How do you envision tools getting an instance of the JavaRuntime for = the CJVMTI dump (the API does not seem to provide a way)? I think the same = mechanism can be used by tools to get one for a process image.=20 JavaRuntime and its constituents seems to be a central part of the API, whi= ch tools will heavily use. So I would go little further and have the API sp= ecify a standard way in which tools can get a JavaRuntime instance for a gi= ven type of artifact. A good metaphor here is, IMO, media files using diffe= rent compression algorithms and codecs which are able to decode them. And a= framework similar to javax.imageio.spi (http://java.sun.com/j2se/1.5.0/doc= s/api/javax/imageio/spi/package-summary.html), but more light-weight could = be developed, so that a tool knowing input artifact type could get a factor= y which can produce a JavaRuntime for it. In the initial RI there could onl= y be one supported artifact type - CJVMTI dump. This way no one can say the= API does not have any implementation. In the future, if someone decides to= write, say, SA-based JavaRuntime provider for core files, tool users would= easily plug the implementation into their tools, provided that the API spe= cifies a consistent algorithm of "JavaRuntime providers" search. Thanks, Konst =20 Intel Novosibirsk Closed Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park,=20 17 Krylatskaya Str., Bldg 4, Moscow 121614,=20 Russian Federation =20 >-----Original Message----- >From: Steve Poole [mailto:spoole167@googlemail.com] >Sent: Saturday, November 28, 2009 10:08 PM >To: kato-spec@incubator.apache.org >Subject: Re: Do we need to change the scope of the API design? > >On Fri, Nov 27, 2009 at 10:35 AM, Bobrovsky, Konstantin S < >konstantin.s.bobrovsky@intel.com> wrote: > >> Steve, >> >> I see what you mean, thank you for the clarifications. I agree that >having >> an RI where a big chunk of the API is unimplemented can be quite >frustrating >> for end users. My only concern is the potential decrease in usability of >> Kato-based products when the Image API is removed. >> When there is an internal VM error such as wrong JIT-generated >code >> leading to a crash, we can not know beforehand what data will be >> required to diagnose the problem. Having all available data (core >> dump) is ideal. >> In the beginning of the JSR many use cases have been developed, and it >> would be interesting to estimate what share of them will be dropped with >> dropping the Image API. What leaves a hope when Image API is gone :-) is >> remaining opportunity for JVM vendors to still produce Kato >implementations >> which can read core dumps and still plug into the Kato-conforming set of >> tools developed by 3rd parties. The loss would be a thick non- >standardized >> layer where external tools developers can't adhere to. >> >> > The runtime restrictions on the SA really effect it's usefulness for >end >> > users of our API. >> >> Are these impossibility to sell tools based on Kato implementation which >> use SA source code or binaries? We have probably already discussed these >> restrictions in the past, but I forgot, sorry. >> >> Its not an impossibility but without the classpath exception we can't >build >an implementation in Apache. The other problem is simply that SA needed >you have have the same OS and JVM as the customer when analysing the dump. > > >> > The cost of writing a corefile reader for hotspot is also >> > quite large - it could easily take someone 6 months to write or more. >> >> That's pretty pessimistic, IMO :) (at least for the part covering Image >> API). 3-5 months would be enough, in my estimation. But that's still big >> chunk of work, I agree. >> >> >I realised that when I talk about dropping the Image API its actually >because we can't implement the Java Runtime API by reading from a core >file. Not because we can't read a corefile. Now my sizing was based on >the >need to implement the whole of the API. The killer cost is the >JavaRuntime >piece. You need to be able to understand the internal layout of the JVM >structures and learn how they stick together etc. > >If we agree that there is value in keeping the Image API around as a means >of reading the contents of a core file without being able to get at Java >Runtime information then we could do so. In that vision we'd just remove >the ability to get a JavaRuntime object from the Image API - ie this >method >( >http://hudson.zones.apache.org/hudson/view/Kato/job/kato.api- >head/javadoc/org/apache/kato/image/ImageProcess.html#getRuntimes%28%29) >would go. > >There are other little changes still required but the general idea seems >quite reasonable. - just comes down to the value of corefile readers >themselves > > What do you think? > > > > >Thanks, >> Konst >> >> >> >-----Original Message----- >> >From: Steve Poole [mailto:spoole167@googlemail.com] >> >Sent: Friday, November 27, 2009 3:34 PM >> >To: kato-spec@incubator.apache.org >> >Subject: Re: Do we need to change the scope of the API design? >> > >> >On Thu, Nov 26, 2009 at 5:02 AM, Bobrovsky, Konstantin S < >> >konstantin.s.bobrovsky@intel.com> wrote: >> > >> >> Hi Steve, >> >> >> >> I think the first option would be ideal of course. But I assume the >> >> initiative to develop RI based on Sun's Serviceability Agent (which >> would >> >> lower the implementation cost dramatically) has failed because of >> >> GPL-related issues. I wish there was a volunteer from the open-source >> who >> >> could adapt SA to the KATO API... >> >> >> >> That would be nice Konst. To be honest though, if I was wishing for >help >> >like this I would prefer to have someone work with us on creating the >> >extended JVMTI or new C API interface as I wrote about previously. >The >> >runtime restrictions on the SA really effect it's usefulness for end >users >> >of our API. The cost of writing a corefile reader for hotspot is >also >> >quite large - it could easily take someone 6 months to write or more. >> >Therefore I'd rather be pragmatic about this and focus on what can be >done >> >by us and what people say they want. >> > >> >From the other two, the second one (remove Image API) IMO would very >much >> >> discord with the original message of the JSR (*postmortem* analysis, >> >which >> >> means digging through the core dumps). So I'd vote for leaving as muc= h >> as >> >> possible of the Image API in the spec, making parts optional as >needed. >> >> >> > >> >I understand about changing the message but I feel uncomfortable to say >> the >> >least to have a large part of our API not supported by the Reference >> >Implementation. I will put together a proposal and explain what I >think >> >needs to be done. >> > >> >You can look at this from anoher point of view. We know that core fil= e >> >sizes are rising ( 8-30GB production heap sizes are now common) and we >> >agreed thereforet at some point core files would become untenable. >> That's >> >why we have the "snapshot" api idea. It seems that for most people th= e >> >usefulness of corefiles is already in doubt so lets focus on producing >a >> >new modern dump mechanism that is fast, scaleable, content configurabl= e >> >and generally cross JVM. Since we will be developing a reference >> >implementation to go with the design we will effectively be producing a >> >modern replacement for HPROF and since the format for this dump is not >> part >> >of the specification (as we have agreed that data formats are always >> >constraining) we will not get rocks thrown at us by JVM vendors who wan= t >> to >> >implement their own alternatives. >> > >> >I think developing the HPROF alternative is the right place to go. >Users >> >will understand the concept and we might actually gain input from >existing >> >HPROF users who want to see improvements made. >> > >> >We are building the current implementation just using JVMTI. Thats not >> >good >> >enough and we need to find ways of getting more data out of a JVM. >Hence >> >the idea of improving JVMTI or adding some new API. That is probabl= y >> >not >> >the right way either but lets discuss.. >> > >> > >> >> Thanks, >> >> Konst >> >> >> >> Intel Novosibirsk >> >> Closed Joint Stock Company Intel A/O >> >> Registered legal address: Krylatsky Hills Business Park, >> >> 17 Krylatskaya Str., Bldg 4, Moscow 121614, >> >> Russian Federation >> >> >> >> >> >> >-----Original Message----- >> >> >From: Steve Poole [mailto:spoole167@googlemail.com] >> >> >Sent: Wednesday, November 25, 2009 7:44 PM >> >> >To: kato-spec@incubator.apache.org >> >> >Subject: Do we need to change the scope of the API design? >> >> > >> >> >Stuart's note on diagnosing JNI problems (thanks for the review >Stuart) >> >> has >> >> >touched on a key point concerning what we want to do and what we ca= n >> >do. >> >> > >> >> >I think its time we discuss where we are and what happens next. >> >> > >> >> >Our desire to specify an API that helps people solve problems has to >be >> >> >tempered with practical implementation considerations. The API >> design >> >> >that IBM contributed to the Kato project is satisfiable by IBM JVMs >(of >> >> >course), but for us to develop a matching reference implementation >we >> >> know >> >> >now that we need to get much more information from the Sun JVM than >we >> >> have >> >> >today. Developing a full solution to match the API as it stands >today >> >is >> >> >difficult for the Kato project as the code we would need to access i= s >> in >> >> >OpenJDK and is under the GPL. >> >> > >> >> >Practically, we have few choices. We could >> >> > >> >> >1 - Get a JVM Vendor to produce , or commit to produce an >> >implementation >> >> >that fully covers the Image API side. >> >> >2 - Remove all the Image API side from the design >> >> >3 - Find a halfway point where some of the Image API stays and is >> >> optional. >> >> > >> >> >I've had many conversations with people concerning option 1 and the >> >> >response >> >> >is generally that its too expensive for the potential return. Both >for >> >> >implementation and ongoing maintenance. >> >> > >> >> >Option 2 seems very draconian - the ability to get at some of the >> >native >> >> >information (such as the native thread for a Java thread) just seems >> too >> >> >useful to remove. >> >> > >> >> >I'd opt for us moving the API design towards option 3 - there are >some >> >> >basic >> >> >changes we could do such as extracting the native access methods int= o >a >> >> >separate interface and allowing the implementer the choice to use it= . >> >> > >> >> >We still need to get access to more data from the JVM (and perhaps i= n >> >> >better >> >> >ways) than we have today. We need to get this data in such a manner >as >> >> not >> >> >to require accessing GPL code and in a way that is cheap to maintain >as >> >> the >> >> >JVM evolves. >> >> > >> >> >My idea for resolving this conundrum is to propose that we include i= n >> >our >> >> >API design either a new JVM 'C' API or extensions to JVMTI. We woul= d >> >not >> >> >implement these additions in the RI but would rely on others to >provide >> >an >> >> >implementation. (We would design the RI and the API to make this >extra >> >> >information optional) >> >> > >> >> >Comments? >> >> > >> >> > >> >> >-- >> >> >Steve >> >> >> > >> > >> > >> >-- >> >Steve >> > > > >-- >Steve