incubator-kato-spec mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stuart Monteith <stuk...@stoo.me.uk>
Subject Re: Do we need to change the scope of the API design?
Date Fri, 27 Nov 2009 17:50:12 GMT
Hi Konstantin,
     Without a proper implementation of the Image API we are missing a 
whole raft of use cases, specifically those to do with
JVM or JNI library diagnosis.

While we might aim to do an implementation of the Kato API that could 
cope with heaps measured in 10s of gigabytes, I don't think the RI 
should. I'd fully expect there to be use cases for smaller JVMs, 
especially those applications that run on desktop machines. Not 
everything is a
enterprise class JEE container. We should make provisions for both the 
small and the large, the smaller JVMs being catered by an full 
implementation that relies on core files. The larger JVM configurations 
should be supported by, perhaps proprietry, partial implementations of 
the API.

Regarding SA. The situation was that we couldn't get the GNU Classpath 
restriction to apply to the SA classes, and it wasn't implemented
as an API at the level we wished to link against. I believe that SA was 
the closest we could get as it had provisions for calling the debugger.
Without support from Sun, using SA is not really an option.

It might be a valid strategy for us to have just the Java API for the 
JSR, which should interest the largest portion of the Java community.
Once it is seen that people are benefiting greatly from it, further 
investment might follow the initial specification and the Image API can
then be specified and implemented in the open.

There are reasons why we won't want to take that approach, we'll have to 
decide what is best on balance.

Regards,
     Stuart

Bobrovsky, Konstantin S wrote:
> Steve,
>
> I see what you mean, thank you for the clarifications. I agree that having an RI where
a big chunk of the API is unimplemented can be quite frustrating for end users. My only concern
is the potential decrease in usability of Kato-based products when the Image API is removed.
> 	When there is an internal VM error such as wrong JIT-generated code
> 	leading to a crash, we can not know beforehand what data will be
> 	required to diagnose the problem. Having all available data (core
> 	dump) is ideal.
> In the beginning of the JSR many use cases have been developed, and it would be interesting
to estimate what share of them will be dropped with dropping the Image API. What leaves a
hope when Image API is gone :-) is remaining opportunity for JVM vendors to still produce
Kato implementations which can read core dumps and still plug into the Kato-conforming set
of tools developed by 3rd parties. The loss would be a thick non-standardized layer where
external tools developers can't adhere to.
>
>    
>> The runtime restrictions on the SA really effect it's usefulness for end
>> users of our API.
>>      
> Are these impossibility to sell tools based on Kato implementation which use SA source
code or binaries? We have probably already discussed these restrictions in the past, but I
forgot, sorry.
>
>    
>> The cost of writing a corefile reader for hotspot is also
>> quite large - it could easily take someone 6 months to write or more.
>>      
> That's pretty pessimistic, IMO :) (at least for the part covering Image API). 3-5 months
would be enough, in my estimation. But that's still big chunk of work, I agree.
>
> Thanks,
> Konst
>
>
>    
>> -----Original Message-----
>> From: Steve Poole [mailto:spoole167@googlemail.com]
>> Sent: Friday, November 27, 2009 3:34 PM
>> To: kato-spec@incubator.apache.org
>> Subject: Re: Do we need to change the scope of the API design?
>>
>> On Thu, Nov 26, 2009 at 5:02 AM, Bobrovsky, Konstantin S<
>> konstantin.s.bobrovsky@intel.com>  wrote:
>>
>>      
>>> Hi Steve,
>>>
>>> I think the first option would be ideal of course. But I assume the
>>> initiative to develop RI based on Sun's Serviceability Agent (which would
>>> lower the implementation cost dramatically) has failed because of
>>> GPL-related issues. I wish there was a volunteer from the open-source who
>>> could adapt SA to the KATO API...
>>>
>>> That would be nice Konst. To be honest though, if I was wishing for help
>>>        
>> like this I would prefer to have someone work with us on creating the
>> extended JVMTI or new C API interface as I wrote about previously.    The
>> runtime restrictions on the SA really effect it's usefulness for end users
>> of our API.     The cost of writing a corefile reader for hotspot is also
>> quite large - it could easily take someone 6 months to write or more.
>> Therefore I'd rather be pragmatic about this and focus on what can be done
>> by us and what people say they want.
>>
>>      
> > From the other two, the second one (remove Image API) IMO would very much
>    
>>> discord with the original message of the JSR (*postmortem* analysis,
>>>        
>> which
>>      
>>> means digging through the core dumps). So I'd vote for leaving as much as
>>> possible of the Image API in the spec, making parts optional as needed.
>>>
>>>        
>> I understand about changing the message but I feel uncomfortable to say the
>> least to have a large part of our API not supported by the Reference
>> Implementation.    I will put together a proposal and explain what I think
>> needs to be done.
>>
>> You can look at this from anoher point of view.   We know that core file
>> sizes are rising ( 8-30GB production heap sizes are now common) and we
>> agreed thereforet at some point core files would become untenable.   That's
>> why we have the "snapshot" api  idea.  It seems that for most people the
>> usefulness of corefiles is already in doubt so  lets focus on producing a
>> new modern dump mechanism that is fast, scaleable,  content configurable
>> and generally cross JVM.   Since we will be developing a reference
>> implementation to go with the design we will effectively be producing a
>> modern replacement for HPROF and since the format for this dump is not part
>> of the specification (as we have agreed that data formats are always
>> constraining) we will not get rocks thrown at us by JVM vendors who want to
>> implement their own alternatives.
>>
>> I think developing the HPROF alternative is the right place to go.  Users
>> will understand the concept and we might actually gain input from existing
>> HPROF users who want to see improvements made.
>>
>> We are building the current implementation just using JVMTI.  Thats not
>> good
>> enough and we need to find ways of getting more data out of a JVM.  Hence
>> the idea of improving JVMTI or adding some new API.     That is probably
>> not
>> the right way either but lets discuss..
>>
>>
>>      
>>> Thanks,
>>> Konst
>>>
>>> Intel Novosibirsk
>>> Closed Joint Stock Company Intel A/O
>>> Registered legal address: Krylatsky Hills Business Park,
>>> 17 Krylatskaya Str., Bldg 4, Moscow 121614,
>>> Russian Federation
>>>
>>>
>>>        
>>>> -----Original Message-----
>>>> From: Steve Poole [mailto:spoole167@googlemail.com]
>>>> Sent: Wednesday, November 25, 2009 7:44 PM
>>>> To: kato-spec@incubator.apache.org
>>>> Subject: Do we need to change the scope of the API design?
>>>>
>>>> Stuart's note on diagnosing JNI problems (thanks for the review Stuart)
>>>>          
>>> has
>>>        
>>>> touched on a key point  concerning what we want to do and what we can
>>>>          
>> do.
>>      
>>>> I think its time we discuss where we are and what happens next.
>>>>
>>>> Our desire to specify an API that helps people solve problems has to be
>>>> tempered with  practical implementation considerations.   The API design
>>>> that IBM contributed to the Kato project is satisfiable by IBM JVMs (of
>>>> course),  but for us to develop a matching reference implementation we
>>>>          
>>> know
>>>        
>>>> now that we need to get much more information from the Sun JVM than we
>>>>          
>>> have
>>>        
>>>> today.  Developing a full solution to match the API as it stands today
>>>>          
>> is
>>      
>>>> difficult for the Kato project as the code we would need to access is in
>>>> OpenJDK and is under the GPL.
>>>>
>>>> Practically, we have few choices.  We could
>>>>
>>>> 1 - Get a JVM Vendor to produce , or commit to produce  an
>>>>          
>> implementation
>>      
>>>> that fully covers the Image API side.
>>>> 2 - Remove all the Image API side from the design
>>>> 3 - Find a halfway point where some of the Image API stays and is
>>>>          
>>> optional.
>>>        
>>>> I've had many conversations with people concerning option 1 and the
>>>> response
>>>> is generally that its too expensive for the potential return.  Both for
>>>> implementation and ongoing maintenance.
>>>>
>>>> Option 2 seems very draconian -  the ability to get at some of the
>>>>          
>> native
>>      
>>>> information (such as the native thread for a Java thread) just seems too
>>>> useful to remove.
>>>>
>>>> I'd opt for us moving the API design towards option 3 - there are some
>>>> basic
>>>> changes we could do such as extracting the native access methods into a
>>>> separate interface and allowing the implementer the choice to use it.
>>>>
>>>> We still need to get access to more data from the JVM (and perhaps in
>>>> better
>>>> ways) than we have today.  We need to get this data in such a manner as
>>>>          
>>> not
>>>        
>>>> to require accessing GPL code and in a way that is cheap to maintain as
>>>>          
>>> the
>>>        
>>>> JVM evolves.
>>>>
>>>> My idea for resolving this conundrum is to propose that we include in
>>>>          
>> our
>>      
>>>> API design either a new JVM 'C' API or  extensions to JVMTI. We would
>>>>          
>> not
>>      
>>>> implement these additions in the RI but would rely on others to provide
>>>>          
>> an
>>      
>>>> implementation.  (We would design the RI and the API to make this extra
>>>> information optional)
>>>>
>>>> Comments?
>>>>
>>>>
>>>> --
>>>> Steve
>>>>          
>>>        
>>
>>
>> --
>> Steve
>>      

-- 
Stuart Monteith
http://blog.stoo.me.uk/


Mime
View raw message