harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Hu" <roberthu...@gmail.com>
Subject Re: [classlib] Code coverage
Date Tue, 21 Aug 2007 10:04:40 GMT
Although I didn't use Cobertura, I found that Cobertura still need use the
"instrumented" classes when testing. It is similar as EMMA.
It's very easy to collect coverage data for a Java application, but very
hard for a JDK, because the tool (EMMA or Cobertura) itself is a Java
program and needs some core Java APIs (some instrumented classes in Harmony
JDK) to initialize. So the biggest problem is that: Java program (include
JUnit test, EMMA or Cobertura) can not initialize successfully on an
"instrumented" JDK (like Harmony). To resolve this problem, we postpone the
instrumentation phase for some kernel classes after the JDK has started
(when the kernel classes are loaded). It's what we did in the customized
EMMA.

So, if you take a look at EMMA's manual, you'll find its working process is
similar with Cobertura. And I think Cobertura will also has that kind of
problem. :-(

2007/8/21, Nathan Beyer <nbeyer@gmail.com>:
>
> I've never used EMMA, so I have no idea. What about Cobertura?
>
> On 8/20/07, Rui Hu <roberthurui@gmail.com> wrote:
> > I think, it is not easy to automatically generate the coverage report of
> our
> > trunk. :-(
> >
> > Because Harmony is not a Java application, it's the JDK, so the original
> > EMMA can not be use to collect the coverage data of Harmony. We've
> modified
> > the source code of EMMA, and realize the coverage collecting for
> core-java
> > APIs. In this way, we must notice that:
> > 1. The coverage of the classes in luni-kernel.jar can not be collected.
> It's
> > not a big problem, because there're not many classes in that jar and the
> > coverage of those classes is considered to be very high (they're usually
> > used kernel classes).
> > 2. The customized EMMA needs extra list of booting classes to run, the
> class
> > list can not be too big or too small. If it's too big, the EMMA may not
> > work. If it's too small, the JDK can not initialize.
> > 3. The coverage of the classes in the list mentioned in #2 will be
> always
> > 100%, (only the class coverage will be 100%, the method/block coverage
> will
> > be correct), it's reasonable for the REAL booting classes, but incorrect
> for
> > the unnecessary classes. So, this is another reason why we need not a
> > too-big list.
> >
> > The problem is from the booting classes list mentioned in #2. Ideally,
> this
> > list includes the classes which is used during initialization of running
> a
> > test suite. Unfortunately, we do not have method to get the list
> > automatically, and it related with the revision of Harmony build, the
> > classes we want to collect coverage data for, and the test suite we want
> to
> > run.
> > So we now must try to get the list from cutting down the big-enough list
> > (usually the all-instrumented-classes-list) every time. If the test
> crash,
> > we must modify the booting classes list or remove some "problem test
> cases".
> >
> >
> > The only shortcut is use the current successful list as the start point,
> but
> > it can not ensure the test will run successfully, especially we change
> the
> > test suite or expand the instrumented classes list ( for example,
> instrument
> > all the classes instead of only public API classes).
> >
> > I'd like to share our customized EMMA and the experience of using it. It
> is
> > highly welcomed to share your idea to automatize this process, or
> simplify
> > the it, or make the it more stably.
> >
> > Thanks in advance !
> >
> > 2007/8/20, Nathan Beyer <nbeyer@gmail.com>:
> > >
> > > A patch for the build scripts would be best, as the coverage could
> > > then be generated at any time by anyone.
> > >
> > > -Nathan
> > >
> > > On 8/19/07, Rui Hu <roberthurui@gmail.com> wrote:
> > > > I've just generated a coverage report on Harmony trunk5 and branch 6
> by
> > > EMMA
> > > > recently. Only public API classes are considered. The EMMA I used
> was
> > > > customized to collect the coverage of core java api.
> > > > The general result is:
> > > > class% --- 68% (1641/2421)
> > > > method% --- 60% (16303/27305)
> > > > block% --- 58% (330724/572138)
> > > >
> > > > The following modules are poor covered , even "zero covered":
> > > > concurrent, CORBA(ORB), XML, RMI, imageio
> > > >
> > > > coverage of awt/swing is not very high, we can improve the whole
> > > coverage
> > > > rapidly by improve these two big module.
> > > >
> > > > BTW,  I need a place to publish the report (folder of many HTMLs),
> any
> > > > advice? Thanks a lot!
> > > >
> > > >
> > > >
> > > > 2007/8/19, Imran Ghory <imranghory@gmail.com>:
> > > > >
> > > > > What the current status with regards to code coverage of the junit
> > > tests ?
> > > > > -ve seen the wiki page but it seems a bit out of date (last
> october).
> > > I've
> > > > > tried getting emma to work myself based upon a few of the mailing
> list
> > > > > posts
> > > > > about it without much luck.
> > > > >
> > > > > Has anyone managed to get an emma instrumented build recently ?
> > > > >
> > > > > Imran
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Robert Hu
> > > > China Software Development Lab, IBM
> > > >
> > >
> >
> >
> >
> > --
> > Robert Hu
> > China Software Development Lab, IBM
> >
>



-- 
Robert Hu
China Software Development Lab, IBM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message