incubator-kato-spec mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Poole <spoole...@googlemail.com>
Subject Re: Do we need to change the scope of the API design?
Date Sat, 28 Nov 2009 16:07:44 GMT
On Fri, Nov 27, 2009 at 10:35 AM, Bobrovsky, Konstantin S <
konstantin.s.bobrovsky@intel.com> wrote:

> Steve,
>
> I see what you mean, thank you for the clarifications. I agree that having
> an RI where a big chunk of the API is unimplemented can be quite frustrating
> for end users. My only concern is the potential decrease in usability of
> Kato-based products when the Image API is removed.
>        When there is an internal VM error such as wrong JIT-generated code
>        leading to a crash, we can not know beforehand what data will be
>        required to diagnose the problem. Having all available data (core
>        dump) is ideal.
> In the beginning of the JSR many use cases have been developed, and it
> would be interesting to estimate what share of them will be dropped with
> dropping the Image API. What leaves a hope when Image API is gone :-) is
> remaining opportunity for JVM vendors to still produce Kato implementations
> which can read core dumps and still plug into the Kato-conforming set of
> tools developed by 3rd parties. The loss would be a thick non-standardized
> layer where external tools developers can't adhere to.
>
> > The runtime restrictions on the SA really effect it's usefulness for end
> > users of our API.
>
> Are these impossibility to sell tools based on Kato implementation which
> use SA source code or binaries? We have probably already discussed these
> restrictions in the past, but I forgot, sorry.
>
> Its not an impossibility but without the classpath exception we can't build
an implementation in Apache.   The other problem is simply that SA needed
you have have the same OS and JVM as the customer when analysing the dump.


> > The cost of writing a corefile reader for hotspot is also
> > quite large - it could easily take someone 6 months to write or more.
>
> That's pretty pessimistic, IMO :) (at least for the part covering Image
> API). 3-5 months would be enough, in my estimation. But that's still big
> chunk of work, I agree.
>
>
I realised that when I talk about dropping the Image API its actually
because we can't implement the Java Runtime API by reading from a core
file.  Not because we can't read a corefile.  Now my sizing was based on the
need to implement the whole of the API.   The killer cost is the JavaRuntime
piece.  You need to be able to understand  the internal layout  of the JVM
structures and learn how they stick together etc.

If we agree that there is value in keeping the Image API around as a means
of  reading the contents of a core file without being able to get at Java
Runtime information then we could do so.  In that vision we'd  just remove
the ability to get a JavaRuntime object from the Image API - ie  this method
(
http://hudson.zones.apache.org/hudson/view/Kato/job/kato.api-head/javadoc/org/apache/kato/image/ImageProcess.html#getRuntimes%28%29)
would go.

There are other little changes still required but the general idea seems
quite reasonable. - just comes down to the value of corefile readers
themselves

 What do you think?




Thanks,
> Konst
>
>
> >-----Original Message-----
> >From: Steve Poole [mailto:spoole167@googlemail.com]
> >Sent: Friday, November 27, 2009 3:34 PM
> >To: kato-spec@incubator.apache.org
> >Subject: Re: Do we need to change the scope of the API design?
> >
> >On Thu, Nov 26, 2009 at 5:02 AM, Bobrovsky, Konstantin S <
> >konstantin.s.bobrovsky@intel.com> wrote:
> >
> >> Hi Steve,
> >>
> >> I think the first option would be ideal of course. But I assume the
> >> initiative to develop RI based on Sun's Serviceability Agent (which
> would
> >> lower the implementation cost dramatically) has failed because of
> >> GPL-related issues. I wish there was a volunteer from the open-source
> who
> >> could adapt SA to the KATO API...
> >>
> >> That would be nice Konst. To be honest though, if I was wishing for help
> >like this I would prefer to have someone work with us on creating the
> >extended JVMTI or new C API interface as I wrote about previously.    The
> >runtime restrictions on the SA really effect it's usefulness for end users
> >of our API.     The cost of writing a corefile reader for hotspot is also
> >quite large - it could easily take someone 6 months to write or more.
> >Therefore I'd rather be pragmatic about this and focus on what can be done
> >by us and what people say they want.
> >
> >From the other two, the second one (remove Image API) IMO would very much
> >> discord with the original message of the JSR (*postmortem* analysis,
> >which
> >> means digging through the core dumps). So I'd vote for leaving as much
> as
> >> possible of the Image API in the spec, making parts optional as needed.
> >>
> >
> >I understand about changing the message but I feel uncomfortable to say
> the
> >least to have a large part of our API not supported by the Reference
> >Implementation.    I will put together a proposal and explain what I think
> >needs to be done.
> >
> >You can look at this from anoher point of view.   We know that core file
> >sizes are rising ( 8-30GB production heap sizes are now common) and we
> >agreed thereforet at some point core files would become untenable.
> That's
> >why we have the "snapshot" api  idea.  It seems that for most people the
> >usefulness of corefiles is already in doubt so  lets focus on producing a
> >new modern dump mechanism that is fast, scaleable,  content configurable
> >and generally cross JVM.   Since we will be developing a reference
> >implementation to go with the design we will effectively be producing a
> >modern replacement for HPROF and since the format for this dump is not
> part
> >of the specification (as we have agreed that data formats are always
> >constraining) we will not get rocks thrown at us by JVM vendors who want
> to
> >implement their own alternatives.
> >
> >I think developing the HPROF alternative is the right place to go.  Users
> >will understand the concept and we might actually gain input from existing
> >HPROF users who want to see improvements made.
> >
> >We are building the current implementation just using JVMTI.  Thats not
> >good
> >enough and we need to find ways of getting more data out of a JVM.  Hence
> >the idea of improving JVMTI or adding some new API.     That is probably
> >not
> >the right way either but lets discuss..
> >
> >
> >> Thanks,
> >> Konst
> >>
> >> Intel Novosibirsk
> >> Closed Joint Stock Company Intel A/O
> >> Registered legal address: Krylatsky Hills Business Park,
> >> 17 Krylatskaya Str., Bldg 4, Moscow 121614,
> >> Russian Federation
> >>
> >>
> >> >-----Original Message-----
> >> >From: Steve Poole [mailto:spoole167@googlemail.com]
> >> >Sent: Wednesday, November 25, 2009 7:44 PM
> >> >To: kato-spec@incubator.apache.org
> >> >Subject: Do we need to change the scope of the API design?
> >> >
> >> >Stuart's note on diagnosing JNI problems (thanks for the review Stuart)
> >> has
> >> >touched on a key point  concerning what we want to do and what we can
> >do.
> >> >
> >> >I think its time we discuss where we are and what happens next.
> >> >
> >> >Our desire to specify an API that helps people solve problems has to be
> >> >tempered with  practical implementation considerations.   The API
> design
> >> >that IBM contributed to the Kato project is satisfiable by IBM JVMs (of
> >> >course),  but for us to develop a matching reference implementation we
> >> know
> >> >now that we need to get much more information from the Sun JVM than we
> >> have
> >> >today.  Developing a full solution to match the API as it stands today
> >is
> >> >difficult for the Kato project as the code we would need to access is
> in
> >> >OpenJDK and is under the GPL.
> >> >
> >> >Practically, we have few choices.  We could
> >> >
> >> >1 - Get a JVM Vendor to produce , or commit to produce  an
> >implementation
> >> >that fully covers the Image API side.
> >> >2 - Remove all the Image API side from the design
> >> >3 - Find a halfway point where some of the Image API stays and is
> >> optional.
> >> >
> >> >I've had many conversations with people concerning option 1 and the
> >> >response
> >> >is generally that its too expensive for the potential return.  Both for
> >> >implementation and ongoing maintenance.
> >> >
> >> >Option 2 seems very draconian -  the ability to get at some of the
> >native
> >> >information (such as the native thread for a Java thread) just seems
> too
> >> >useful to remove.
> >> >
> >> >I'd opt for us moving the API design towards option 3 - there are some
> >> >basic
> >> >changes we could do such as extracting the native access methods into a
> >> >separate interface and allowing the implementer the choice to use it.
> >> >
> >> >We still need to get access to more data from the JVM (and perhaps in
> >> >better
> >> >ways) than we have today.  We need to get this data in such a manner as
> >> not
> >> >to require accessing GPL code and in a way that is cheap to maintain as
> >> the
> >> >JVM evolves.
> >> >
> >> >My idea for resolving this conundrum is to propose that we include in
> >our
> >> >API design either a new JVM 'C' API or  extensions to JVMTI. We would
> >not
> >> >implement these additions in the RI but would rely on others to provide
> >an
> >> >implementation.  (We would design the RI and the API to make this extra
> >> >information optional)
> >> >
> >> >Comments?
> >> >
> >> >
> >> >--
> >> >Steve
> >>
> >
> >
> >
> >--
> >Steve
>



-- 
Steve

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message