incubator-kato-spec mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bobrovsky, Konstantin S" <konstantin.s.bobrov...@intel.com>
Subject RE: Do we need to change the scope of the API design?
Date Tue, 01 Dec 2009 11:07:57 GMT
Hi Steve, all,

>I realised that when I talk about dropping the Image API its actually
>because we can't implement the Java Runtime API by reading from a core
>file.  Not because we can't read a corefile.  Now my sizing was based on
>the
>need to implement the whole of the API.   The killer cost is the
>JavaRuntime
>piece.  You need to be able to understand  the internal layout  of the JVM
>structures and learn how they stick together etc.
>
>If we agree that there is value in keeping the Image API around as a means
>of  reading the contents of a core file without being able to get at Java
>Runtime information then we could do so.  In that vision we'd  just remove
>the ability to get a JavaRuntime object from the Image API - ie  this
>method
>(
>http://hudson.zones.apache.org/hudson/view/Kato/job/kato.api-
>head/javadoc/org/apache/kato/image/ImageProcess.html#getRuntimes%28%29)
>would go.
>
>There are other little changes still required but the general idea seems
>quite reasonable. - just comes down to the value of corefile readers
>themselves
>
> What do you think?

I think this is a good idea. Process-level core file analysis sometimes helps too and it is
good to standardize such things as ImageRegister/Process/Module/Section, etc.

Instead of ImageProcess.getRuntimes, the task of (future) getting a JavaRuntime for a core
file by a 3rd party tool can probably be solved in another way.

AFAIU, current RI uses an output from the CJVMTI agent (HPROF alternative) as the source artifact
for building a JavaRuntime based on it - is that correct? How do you envision tools getting
an instance of the JavaRuntime for the CJVMTI dump (the API does not seem to provide a way)?
I think the same mechanism can be used by tools to get one for a process image. 

JavaRuntime and its constituents seems to be a central part of the API, which tools will heavily
use. So I would go little further and have the API specify a standard way in which tools can
get a JavaRuntime instance for a given type of artifact. A good metaphor here is, IMO, media
files using different compression algorithms and codecs which are able to decode them. And
a framework similar to javax.imageio.spi (http://java.sun.com/j2se/1.5.0/docs/api/javax/imageio/spi/package-summary.html),
but more light-weight could be developed, so that a tool knowing input artifact type could
get a factory which can produce a JavaRuntime for it. In the initial RI there could only be
one supported artifact type - CJVMTI dump. This way no one can say the API does not have any
implementation. In the future, if someone decides to write, say, SA-based JavaRuntime provider
for core files, tool users would easily plug the implementation into their tools, provided
that the API specifies a consistent algorithm of "JavaRuntime providers" search.

Thanks,
Konst
 
Intel Novosibirsk
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park, 
17 Krylatskaya Str., Bldg 4, Moscow 121614, 
Russian Federation
 

>-----Original Message-----
>From: Steve Poole [mailto:spoole167@googlemail.com]
>Sent: Saturday, November 28, 2009 10:08 PM
>To: kato-spec@incubator.apache.org
>Subject: Re: Do we need to change the scope of the API design?
>
>On Fri, Nov 27, 2009 at 10:35 AM, Bobrovsky, Konstantin S <
>konstantin.s.bobrovsky@intel.com> wrote:
>
>> Steve,
>>
>> I see what you mean, thank you for the clarifications. I agree that
>having
>> an RI where a big chunk of the API is unimplemented can be quite
>frustrating
>> for end users. My only concern is the potential decrease in usability of
>> Kato-based products when the Image API is removed.
>>        When there is an internal VM error such as wrong JIT-generated
>code
>>        leading to a crash, we can not know beforehand what data will be
>>        required to diagnose the problem. Having all available data (core
>>        dump) is ideal.
>> In the beginning of the JSR many use cases have been developed, and it
>> would be interesting to estimate what share of them will be dropped with
>> dropping the Image API. What leaves a hope when Image API is gone :-) is
>> remaining opportunity for JVM vendors to still produce Kato
>implementations
>> which can read core dumps and still plug into the Kato-conforming set of
>> tools developed by 3rd parties. The loss would be a thick non-
>standardized
>> layer where external tools developers can't adhere to.
>>
>> > The runtime restrictions on the SA really effect it's usefulness for
>end
>> > users of our API.
>>
>> Are these impossibility to sell tools based on Kato implementation which
>> use SA source code or binaries? We have probably already discussed these
>> restrictions in the past, but I forgot, sorry.
>>
>> Its not an impossibility but without the classpath exception we can't
>build
>an implementation in Apache.   The other problem is simply that SA needed
>you have have the same OS and JVM as the customer when analysing the dump.
>
>
>> > The cost of writing a corefile reader for hotspot is also
>> > quite large - it could easily take someone 6 months to write or more.
>>
>> That's pretty pessimistic, IMO :) (at least for the part covering Image
>> API). 3-5 months would be enough, in my estimation. But that's still big
>> chunk of work, I agree.
>>
>>
>I realised that when I talk about dropping the Image API its actually
>because we can't implement the Java Runtime API by reading from a core
>file.  Not because we can't read a corefile.  Now my sizing was based on
>the
>need to implement the whole of the API.   The killer cost is the
>JavaRuntime
>piece.  You need to be able to understand  the internal layout  of the JVM
>structures and learn how they stick together etc.
>
>If we agree that there is value in keeping the Image API around as a means
>of  reading the contents of a core file without being able to get at Java
>Runtime information then we could do so.  In that vision we'd  just remove
>the ability to get a JavaRuntime object from the Image API - ie  this
>method
>(
>http://hudson.zones.apache.org/hudson/view/Kato/job/kato.api-
>head/javadoc/org/apache/kato/image/ImageProcess.html#getRuntimes%28%29)
>would go.
>
>There are other little changes still required but the general idea seems
>quite reasonable. - just comes down to the value of corefile readers
>themselves
>
> What do you think?
>
>
>
>
>Thanks,
>> Konst
>>
>>
>> >-----Original Message-----
>> >From: Steve Poole [mailto:spoole167@googlemail.com]
>> >Sent: Friday, November 27, 2009 3:34 PM
>> >To: kato-spec@incubator.apache.org
>> >Subject: Re: Do we need to change the scope of the API design?
>> >
>> >On Thu, Nov 26, 2009 at 5:02 AM, Bobrovsky, Konstantin S <
>> >konstantin.s.bobrovsky@intel.com> wrote:
>> >
>> >> Hi Steve,
>> >>
>> >> I think the first option would be ideal of course. But I assume the
>> >> initiative to develop RI based on Sun's Serviceability Agent (which
>> would
>> >> lower the implementation cost dramatically) has failed because of
>> >> GPL-related issues. I wish there was a volunteer from the open-source
>> who
>> >> could adapt SA to the KATO API...
>> >>
>> >> That would be nice Konst. To be honest though, if I was wishing for
>help
>> >like this I would prefer to have someone work with us on creating the
>> >extended JVMTI or new C API interface as I wrote about previously.
>The
>> >runtime restrictions on the SA really effect it's usefulness for end
>users
>> >of our API.     The cost of writing a corefile reader for hotspot is
>also
>> >quite large - it could easily take someone 6 months to write or more.
>> >Therefore I'd rather be pragmatic about this and focus on what can be
>done
>> >by us and what people say they want.
>> >
>> >From the other two, the second one (remove Image API) IMO would very
>much
>> >> discord with the original message of the JSR (*postmortem* analysis,
>> >which
>> >> means digging through the core dumps). So I'd vote for leaving as much
>> as
>> >> possible of the Image API in the spec, making parts optional as
>needed.
>> >>
>> >
>> >I understand about changing the message but I feel uncomfortable to say
>> the
>> >least to have a large part of our API not supported by the Reference
>> >Implementation.    I will put together a proposal and explain what I
>think
>> >needs to be done.
>> >
>> >You can look at this from anoher point of view.   We know that core file
>> >sizes are rising ( 8-30GB production heap sizes are now common) and we
>> >agreed thereforet at some point core files would become untenable.
>> That's
>> >why we have the "snapshot" api  idea.  It seems that for most people the
>> >usefulness of corefiles is already in doubt so  lets focus on producing
>a
>> >new modern dump mechanism that is fast, scaleable,  content configurable
>> >and generally cross JVM.   Since we will be developing a reference
>> >implementation to go with the design we will effectively be producing a
>> >modern replacement for HPROF and since the format for this dump is not
>> part
>> >of the specification (as we have agreed that data formats are always
>> >constraining) we will not get rocks thrown at us by JVM vendors who want
>> to
>> >implement their own alternatives.
>> >
>> >I think developing the HPROF alternative is the right place to go.
>Users
>> >will understand the concept and we might actually gain input from
>existing
>> >HPROF users who want to see improvements made.
>> >
>> >We are building the current implementation just using JVMTI.  Thats not
>> >good
>> >enough and we need to find ways of getting more data out of a JVM.
>Hence
>> >the idea of improving JVMTI or adding some new API.     That is probably
>> >not
>> >the right way either but lets discuss..
>> >
>> >
>> >> Thanks,
>> >> Konst
>> >>
>> >> Intel Novosibirsk
>> >> Closed Joint Stock Company Intel A/O
>> >> Registered legal address: Krylatsky Hills Business Park,
>> >> 17 Krylatskaya Str., Bldg 4, Moscow 121614,
>> >> Russian Federation
>> >>
>> >>
>> >> >-----Original Message-----
>> >> >From: Steve Poole [mailto:spoole167@googlemail.com]
>> >> >Sent: Wednesday, November 25, 2009 7:44 PM
>> >> >To: kato-spec@incubator.apache.org
>> >> >Subject: Do we need to change the scope of the API design?
>> >> >
>> >> >Stuart's note on diagnosing JNI problems (thanks for the review
>Stuart)
>> >> has
>> >> >touched on a key point  concerning what we want to do and what we can
>> >do.
>> >> >
>> >> >I think its time we discuss where we are and what happens next.
>> >> >
>> >> >Our desire to specify an API that helps people solve problems has to
>be
>> >> >tempered with  practical implementation considerations.   The API
>> design
>> >> >that IBM contributed to the Kato project is satisfiable by IBM JVMs
>(of
>> >> >course),  but for us to develop a matching reference implementation
>we
>> >> know
>> >> >now that we need to get much more information from the Sun JVM than
>we
>> >> have
>> >> >today.  Developing a full solution to match the API as it stands
>today
>> >is
>> >> >difficult for the Kato project as the code we would need to access is
>> in
>> >> >OpenJDK and is under the GPL.
>> >> >
>> >> >Practically, we have few choices.  We could
>> >> >
>> >> >1 - Get a JVM Vendor to produce , or commit to produce  an
>> >implementation
>> >> >that fully covers the Image API side.
>> >> >2 - Remove all the Image API side from the design
>> >> >3 - Find a halfway point where some of the Image API stays and is
>> >> optional.
>> >> >
>> >> >I've had many conversations with people concerning option 1 and the
>> >> >response
>> >> >is generally that its too expensive for the potential return.  Both
>for
>> >> >implementation and ongoing maintenance.
>> >> >
>> >> >Option 2 seems very draconian -  the ability to get at some of the
>> >native
>> >> >information (such as the native thread for a Java thread) just seems
>> too
>> >> >useful to remove.
>> >> >
>> >> >I'd opt for us moving the API design towards option 3 - there are
>some
>> >> >basic
>> >> >changes we could do such as extracting the native access methods into
>a
>> >> >separate interface and allowing the implementer the choice to use it.
>> >> >
>> >> >We still need to get access to more data from the JVM (and perhaps in
>> >> >better
>> >> >ways) than we have today.  We need to get this data in such a manner
>as
>> >> not
>> >> >to require accessing GPL code and in a way that is cheap to maintain
>as
>> >> the
>> >> >JVM evolves.
>> >> >
>> >> >My idea for resolving this conundrum is to propose that we include in
>> >our
>> >> >API design either a new JVM 'C' API or  extensions to JVMTI. We would
>> >not
>> >> >implement these additions in the RI but would rely on others to
>provide
>> >an
>> >> >implementation.  (We would design the RI and the API to make this
>extra
>> >> >information optional)
>> >> >
>> >> >Comments?
>> >> >
>> >> >
>> >> >--
>> >> >Steve
>> >>
>> >
>> >
>> >
>> >--
>> >Steve
>>
>
>
>
>--
>Steve

Mime
View raw message