harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Beyer" <nbe...@kc.rr.com>
Subject RE: [classlib] [testing] Coverage (was Re: 37% of total test execution time is spent in a single test)
Date Wed, 14 Jun 2006 00:06:48 GMT
Is there any possibility of adding this to an Ant script that can optionally
be run with the build scripts? I'd like to make this easier for everyone to
run while developing.

Has anyone tried any other coverage tools? Like TPTP for Eclipse, Clover or
Corbetura (sp)?

-Nathan

> -----Original Message-----
> From: Vladimir Ivanov [mailto:ivavladimir@gmail.com]
> 
> Latest Harmony API source coverage by Harmony API unit tests results I
> stored at wiki page
> 
> http://wiki.apache.org/harmony/Coverage_information
> 
> 
> 
> I'm going to refresh it bi-weekly (seems, it is enough for coverage).
> 
> 
> 
> >I think we have got a agreement on the test naming convention[1], but
> >for sure, there have been many (legacy) test cases before this
> >agreement, and I think the volunteer is highly welcome to provide patch
> >for them.
> >
> >[1]http://incubator.apache.org/harmony/subcomponents/classlibrary/testing
> .html
> 
> >
> 
> If nobody objects I'm going to look through the unit tests to correct
> package names according to the agreement (where needed).
> 
> 
> 
>  Thanks,
> 
>    Vladimir
> 
> 
> 
> On 6/8/06, Paulex Yang <paulex.yang@gmail.com > wrote:
> >
> > Vladimir,
> >
> > Vladimir Ivanov wrote:
> > > Thanks Paulex!
> > >
> > > I did the same, but could not send results due to spam filter J
> > > Observations:
> > >
> > >   1. Coverage results look pretty much similar.
> > >   2. Exclude list looks pretty much similar too, but, looks like it
> > >   depends on the way of data collection (I didn't run ant task and the
> > > list is
> > >   a little bit different).
> > Great.
> > >
> > > In any case, I think, when we run harmony on another VM exclude list
> > will
> > > have to be updated.
> > >
> > >
> > >
> > > May be we can start publishing the coverage information on wiki pages
> > and
> > > provide some updates time to time (I can do it)?
> > >
> > +1,  and of course, you can only if  no one in the mailing list objects
> > and you'll have my welcome, and I think it will be even greater if these
> > reports can be generated regularly like what JAPI is doing:)
> > >
> > >
> > > One note:
> > >
> > > I noticed that different unit tests have very different package names
> > >
> > > Now the directory with all built tests copied to one place looks like:
> > I think we have got a agreement on the test naming convention[1], but
> > for sure, there have been many (legacy) test cases before this
> > agreement, and I think the volunteer is highly welcome to provide patch
> > for them.
> >
> >
> [1]http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.
> html
> >
> > >
> > >
> > >
> > > C:\coverage\tests\test>ls
> > >
> > > GZIPOutClose2.txt        api           config         javax
> > > tests
> > >
> > > GZIPOutFinish.txt        api.injected  dazzle         org
> xml
> >
> > >
> > > GZIPOutWrite.txt         binary        gif            prefs
> > >
> > > Inet6Address.golden.ser  bundles       impl           serialization
> > >
> > > JDK2-3gabba.zip          com           impl.injected   test.txt
> > >
> > >
> > >
> > > I think, it would be good if tests had unified package names.
> > >
> > > Why? - so far, just common sense, just to have an order in test suite
> > >
> > > Organization (if consider all unit tests as solid test suite).
> > >
> > > Thanks,
> > >  Vladimir
> > >
> > > For example, my exclude list for java.io is:
> > > -java.io.BufferedInputStream ,
> > > -java.io.BufferedOutputStream,
> > > -java.io.File ,
> > > -java.io.FileChannelFactory,
> > > -java.io.FileDescriptor,
> > > -java.io.FileInputStream,
> > > -java.io.FileOutputStream,
> > > -java.io.FilterInputStream ,
> > > -java.io.FilterOutputStream,
> > > - java.io.InputStream,
> > > -java.io.OutputStream,
> > > -java.io.ObjectStreamField,
> > > -java.io.PrintStream
> > >
> > >
> > > On 6/6/06, Paulex Yang < paulex.yang@gmail.com > wrote:
> > >>
> > >> I've attach the scripts and excluded class lists to JIRA,  please
> refer
> > >> to https://issues.apache.org/jira/browse/HARMONY-564 .  Enjoy it:).
> > >>
> > >> Mark Hindess wrote:
> > >> > On 2 June 2006 at 10:37, Paulex Yang < paulex.yang@gmail.com>
> wrote:
> > >> >
> > >> >> Mark,
> > >> >>
> > >> >> I'm glad that there is someone else has interest on emma, I've
> > >> tried it
> > >> >> before. AFAIK, emma works by instrumentation, but sometimes for
> > >> classes
> > >> >> in bootclasspath, the instrumentation cannot work, there are two
> > >> cases:
> > >>
> > >> >> 1. Some instrumented classes cannot be loaded by VM.
> > >> >> 2. Some classes cannot be instrumented
> > >> >>
> > >> >> I have tried to look more inside to find some way to work around
> > >> but I
> > >> >> haven't got enough time yet.
> > >> >>
> > >> >> Specifically for nio-channel module, I had a list for these two
> > cases
> > >> (I
> > >> >> believe the data is a little outdated and should be reevaluated)
> > >> >> case 1.
> > >> >> BaseByteBuffer.class
> > >> >> Buffer.class
> > >> >> BufferFactory.class
> > >> >> ByteBuffer.class
> > >> >> CharArrayBuffer.class
> > >> >> CharBuffer.class
> > >> >> HeapByteBuffer.class
> > >> >> ReadWriteCharArrayBuffer.class
> > >> >> ReadWriteHeapByteBuffer.class
> > >> >> FileChannel.class
> > >> >> AbstractInterruptibleChannel.class
> > >> >> FileChannelImpl.class
> > >> >> WriteOnlyFileChannel.class
> > >> >> LockManager.class
> > >> >> LockManager$1.class
> > >> >> ReadOnlyFileChannel.class
> > >> >>
> > >> >> case 2:
> > >> >> ByteChannel.class
> > >> >> Channel.class
> > >> >> GatheringByteChannel.class
> > >> >> InterruptibleChannel.class
> > >> >> WritableByteChannel.class
> > >> >>
> > >> >> And I have got some ant script and more excluded list for emma,
if
> > >> >> anyone has interests, I can upload it to JIRA.
> > >> >>
> > >> >
> > >> > Yes!
> > >> >
> > >> > -Mark.
> > >> >
> > >> >
> > >> >> Mark Hindess wrote:
> > >> >>
> > >> >>> Anyone tried using emma (emma.sf.net) to look at test coverage
> > >> for our
> > >> >>> tests?  That might help ensure we don't trim them down too
much?
> > >> >>>
> > >> >>> I suspect we will soon reach a point where there are
> significantly
> > >> fewer
> > >> >>> obvious opportunities for small contributions - e.g.
> > >> 1.5generification,
> > >> >>> etc from looking at JAPI reports.  Having test coverage reports
> > >> might
> > >> >>> help by highlighting areas for improvement.
> > >> >>>
> > >> >>> I had a quick try with:
> > >> >>>
> > >> >>>   ant -f make/build.xml
> > >> >>>   ant -f modules/nio/make/build.xml test
> > >> >>>   java -cp emma.jar emma instr -m overwrite -cp
> > >> deploy/jdk/jre/lib/boot/nio
> > >> >>>
> > >> >> .jar
> > >> >>
> > >> >>> and then:
> > >> >>>
> > >> >>>   java -Xbootclasspath/p: emma.jar \
> > >> >>>        -cp build/tests:modules/nio/bin/test:junit.jar:. \
> > >> >>>        junit.textui.TestRunner \
> > >> >>>
> org.apache.harmony.tests.java.nio.channels.SocketChannelTest
> > >> >>>
> > >> >>> (Running all the tests is a little tricky because test depends
on
> > >> build,
> > >> >>> which depends on clean which wipes out the instr'd jar. ;-(
)
> > >> >>>
> > >> >>> Sadly it dies horribly with the stack trace below.  I'll take
> > >> another
> > >> >>> look when I have a little more time, but I thought I'd mention
it
> > in
> > >> >>> case anyone else had any thoughts on the subject.
> > >> >>>
> > >> >>> Regards,
> > >> >>>
> > >> >>> Mark.
> > >> >>>
> > >> >>> Exception in thread "main" java/lang/ExceptionInInitializerError
> > >> >>>         at java/lang/J9VMInternals.initialize
> > >> (J9VMInternals.java:203)
> > >> >>>         at java/nio/Buffer.$VRi ( Buffer.java:4294967295)
> > >> >>>         at java/nio/Buffer.<clinit> ( Buffer.java:4294967295)
> > >> >>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> > >> >>>         at java/lang/J9VMInternals.initialize (
> J9VMInternals.java
> > >> :185)
> > >> >>>         at java/lang/J9VMInternals.initialize
> > >> (J9VMInternals.java:154)
> > >> >>>         at java/lang/String.getBytes (String.java:783)
> > >> >>>         at org/apache/harmony/luni/util/Util.getBytes (
> > >> Util.java:54)
> > >> >>>         at java/lang/ClassLoader.loadLibraryWithPath
> > >> (ClassLoader.java
> > >> :998)
> > >> >>>         at java/lang/ClassLoader.loadLibraryWithClassLoader
(
> > >> ClassLoader.ja
> > >> >>>
> > >> >> va:970)
> > >> >>
> > >> >>>         at java/lang/System.loadLibrary (System.java:706)
> > >> >>>         at com/ibm/icu4jni/common/ICU4JNILoader.loadLibrary
> > (Unknown
> > >> Source
> > >> >>>
> > >> >> )
> > >> >>
> > >> >>>         at com/ibm/icu4jni/converters/NativeConverter.<clinit>
> > >> (Unknown Sou
> > >> >>>
> > >> >> rce)
> > >> >>
> > >> >>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> > >> >>>         at java/lang/J9VMInternals.initialize
> > >> ( J9VMInternals.java:185)
> > >> >>>         at
> > com/ibm/icu4jni/charset/CharsetProviderICU.charsetForName
> > >> (Unkno
> > >> >>>
> > >> >> wn Source)
> > >> >>
> > >> >>>         at java/nio/charset/Charset.forNameInternal
> > >> (Charset.java:450)
> > >> >>>         at java/nio/charset/Charset.forName (Charset.java:513)
> > >> >>>         at java/lang/String.defaultCharset ( String.java:688)
> > >> >>>         at java/lang/String.<init> ( String.java:252)
> > >> >>>         at org/apache/harmony/luni/util/Util.toString (Util.java
> > :74)
> > >> >>>         at java/lang/System.getPropertyList (Native Method)
> > >> >>>         at java/lang/System.ensureProperties ( System.java:496)
> > >> >>>         at java/lang/System.<clinit> (System.java:55)
> > >> >>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> > >> >>>         at java/lang/J9VMInternals.initialize
> > >> (J9VMInternals.java:185)
> > >> >>>         at java/lang/ClassLoader.initializeClassLoaders (
> > >> ClassLoader.java:6
> > >> >>>
> > >> >> 5)
> > >> >>
> > >> >>>         at java/lang/Thread.initialize (Thread.java:301)
> > >> >>>         at java/lang/Thread.<init> (Thread.java:120)
> > >> >>> java/lang/NullPointerException
> > >> >>>         at java/lang/Throwable.printStackTrace ( Throwable.java
> > :231)
> > >> >>>         at
> com/vladium/emma/rt/RTExitHook.createClassLoaderClosure
> > >> (RTExitH
> > >> >>>
> > >> >> ook.java:63)
> > >> >>
> > >> >>>         at com/vladium/emma/rt/RT.reset ( RT.java:92)
> > >> >>>         at com/vladium/emma/rt/RT.<clinit> (RT.java:238)
> > >> >>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> > >> >>>         at java/lang/J9VMInternals.initialize (
> J9VMInternals.java
> > >> :185)
> > >> >>>         at java/nio/Buffer.$VRi ( Buffer.java:4294967295)
> > >> >>>         at java/nio/Buffer.<clinit> (Buffer.java:4294967295)
> > >> >>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> > >> >>>         at java/lang/J9VMInternals.initialize
> > >> (J9VMInternals.java:185)
> > >> >>>         at java/lang/J9VMInternals.initialize
> > >> (J9VMInternals.java:154)
> > >> >>>         at java/lang/String.getBytes ( String.java :783)
> > >> >>>         at org/apache/harmony/luni/util/Util.getBytes (Util.java
> > :54)
> > >> >>>         at java/lang/ClassLoader.loadLibraryWithPath
> > >> (ClassLoader.java
> > >> :998)
> > >> >>>         at java/lang/ClassLoader.loadLibraryWithClassLoader
(
> > >> ClassLoader.ja
> > >> >>>
> > >> >> va:970)
> > >> >>
> > >> >>>         at java/lang/System.loadLibrary ( System.java:706)
> > >> >>>         at com/ibm/icu4jni/common/ICU4JNILoader.loadLibrary
> > (Unknown
> > >> Source
> > >> >>>
> > >> >> )
> > >> >>
> > >> >>>         at com/ibm/icu4jni/converters/NativeConverter.<clinit>
> > >> (Unknown Sou
> > >> >>>
> > >> >> rce)
> > >> >>
> > >> >>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> > >> >>>         at java/lang/J9VMInternals.initialize
> > >> (J9VMInternals.java:185)
> > >> >>>         at
> > com/ibm/icu4jni/charset/CharsetProviderICU.charsetForName
> > >> (Unkno
> > >> >>>
> > >> >> wn Source)
> > >> >>
> > >> >>>         at java/nio/charset/Charset.forNameInternal
> > >> (Charset.java:450)
> > >> >>>         at java/nio/charset/Charset.forName (Charset.java:513)
> > >> >>>         at java/lang/String.defaultCharset ( String.java:688)
> > >> >>>         at java/lang/String.<init> (String.java:252)
> > >> >>>         at org/apache/harmony/luni/util/Util.toString (Util.java
> > :74)
> > >> >>>         at java/lang/System.getPropertyList (Native Method)
> > >> >>>         at java/lang/System.ensureProperties (System.java:496)
> > >> >>>         at java/lang/System.<clinit> (System.java:55)
> > >> >>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> > >> >>>         at java/lang/J9VMInternals.initialize
> > >> (J9VMInternals.java:185)
> > >> >>>         at java/lang/ClassLoader.initializeClassLoaders (
> > >> ClassLoader.java:6
> > >> >>>
> > >> >> 5)
> > >> >>
> > >> >>>         at java/lang/Thread.initialize (Thread.java:301)
> > >> >>>         at java/lang/Thread.<init> (Thread.java :120)
> > >> >>> JVMJ9VM015W Initialization error for library jclclear_23(14):
> > >> JVMJ9VM009E J
> > >> >>>
> > >> >> 9VMDllMain failed
> > >> >>
> > >> >>> HMYEXEL062E Internal VM error: Failed to create Java VM
> > >> >>> FAILED.
> > >> >>>
> > >> >>>
> > >> >>> On 1 June 2006 at 22:26, "Mikhail Loenko" < mloenko@gmail.com
>
> > >> wrote:
> > >> >>>
> > >> >>>
> > >> >>>> 2006/6/1, Geir Magnusson Jr < geir@pobox.com>:
> > >> >>>>
> > >> >>>>
> > >> >>>>> Mikhail Loenko wrote:
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>> 2006/6/1, Geir Magnusson Jr < geir@pobox.com
>:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>> I don't quite grok what you mean - could you
post what you
> are
> > >> >>>>>>> suggesting in code here on -dev?
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>> Index:
> > >> modules/luni/src/test/java/tests/api/java/net/Inet6AddressTest.ja
> > >> >>>>>>
> > >> >> v
> > >> >>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>> a
> > >> >>>>
> > >> >>>>
> > >> >>>>>>
> > >> ===================================================================
> > >> >>>>>> ---
> > >> >>>>>>
> > >> modules/luni/src/test/java/tests/api/java/net/Inet6AddressTest.java
> > >> >>>>>> (revision
> > >> >>>>>> 410480)
> > >> >>>>>> +++
> > >> >>>>>>
> > >> modules/luni/src/test/java/tests/api/java/net/Inet6AddressTest.java
> > >>
> > >> >>>>>> (working
> > >> >>>>>> copy)
> > >> >>>>>> @@ -785,7 +785,7 @@
> > >> >>>>>>                 "FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:
> 255.255.255.255
> > ",
> > >> >>>>>>                 "0:0:0:0:0:0:0:0", "0:0:0:0:0:0:
0.0.0.0" };
> > >> >>>>>>
> > >> >>>>>> -        String invalidIPAddresses[] = { ":",
"FFFF:FFFF",
> > "1:1",
> > >> >>>>>> "::1.2.3.444" };
> > >> >>>>>> +        String invalidIPAddresses[] = { "FFFF:FFFF"
};
> > >> >>>>>>
> > >> >>>>>>         for (int i = 0; i < validIPAddresses.length;
i++) {
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>> Thx - why do you want to remove testing the others?
 Because
> you
> > >> believe
> > >> >>>>> that it is in effect testing the resolver in the OS
since each
> > >> returns
> > >> >>>>> the same answer,and thus it's testing the same thing
over and
> > >> over?
> > >> >>>>>
> > >> >>>>>
> > >> >>>> I believe so, but I'd like to hear confirmation from code
> authors
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> Mikhail
> > >> >>>>
> > >> >>>>
> > >> >>>
> > >> >>>
> > >> ---------------------------------------------------------------------
> > >> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> >>> To unsubscribe, e-mail: harmony-dev-
> unsubscribe@incubator.apache.org
> >
> > >> >>> For additional commands, e-mail:
> > >> harmony-dev-help@incubator.apache.org
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >> --
> > >> >> Paulex Yang
> > >> >> China Software Development Lab
> > >> >> IBM
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > ---------------------------------------------------------------------
> > >> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> >> To unsubscribe, e-mail: harmony-dev-
> unsubscribe@incubator.apache.org
> > >> >> For additional commands, e-mail:
> > >> harmony-dev-help@incubator.apache.org
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > -------------------------------------------------------------------
> --
> >
> > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> > To unsubscribe, e-mail: harmony-dev-
> unsubscribe@incubator.apache.org
> > >> > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > >> >
> > >> >
> > >> >
> > >>
> > >>
> > >> --
> > >> Paulex Yang
> > >> China Software Development Lab
> > >> IBM
> > >>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > >> For additional commands, e-mail: harmony-dev-
> help@incubator.apache.org
> > >>
> > >>
> > >
> >
> >
> > --
> > Paulex Yang
> > China Software Development Lab
> > IBM
> >
> >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message