harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hindess <mark.hind...@googlemail.com>
Subject Re: [classlib] [testing] Coverage (was Re: 37% of total test execution time is spent in a single test)
Date Wed, 14 Jun 2006 13:15:36 GMT

On 14 June 2006 at 7:04, Geir Magnusson Jr <geir@pobox.com> wrote:
>
> yes.
> 
> And/or add it to the upcoming donation of the [testing] infrastructure
> to make it part of the CI runs.

Are you talking about me?  It's just an ant file and a few script so I
hope you aren't expecting too much.

> geir
> 
> Nathan Beyer wrote:
> >
> > Is there any possibility of adding this to an Ant script that can
> > optionally be run with the build scripts? I'd like to make this
> > easier for everyone to run while developing.

Nathan,

I'm look at doing exactly this.

Regards,
 Mark.

> > Has anyone tried any other coverage tools? Like TPTP for Eclipse, Clover or
> > Corbetura (sp)?
> > 
> > -Nathan
> > 
> >> -----Original Message-----
> >> From: Vladimir Ivanov [mailto:ivavladimir@gmail.com]
> >>
> >> Latest Harmony API source coverage by Harmony API unit tests results I
> >> stored at wiki page
> >>
> >> http://wiki.apache.org/harmony/Coverage_information
> >>
> >>
> >>
> >> I'm going to refresh it bi-weekly (seems, it is enough for coverage).
> >>
> >>
> >>
> >>> I think we have got a agreement on the test naming convention[1], but
> >>> for sure, there have been many (legacy) test cases before this
> >>> agreement, and I think the volunteer is highly welcome to provide patch
> >>> for them.
> >>>
> >>> [1]http://incubator.apache.org/harmony/subcomponents/classlibrary/testing
> >> .html
> >>
> >> If nobody objects I'm going to look through the unit tests to correct
> >> package names according to the agreement (where needed).
> >>
> >>
> >>
> >>  Thanks,
> >>
> >>    Vladimir
> >>
> >>
> >>
> >> On 6/8/06, Paulex Yang <paulex.yang@gmail.com > wrote:
> >>> Vladimir,
> >>>
> >>> Vladimir Ivanov wrote:
> >>>> Thanks Paulex!
> >>>>
> >>>> I did the same, but could not send results due to spam filter J
> >>>> Observations:
> >>>>
> >>>>   1. Coverage results look pretty much similar.
> >>>>   2. Exclude list looks pretty much similar too, but, looks like it
> >>>>   depends on the way of data collection (I didn't run ant task and the
> >>>> list is
> >>>>   a little bit different).
> >>> Great.
> >>>> In any case, I think, when we run harmony on another VM exclude list
> >>> will
> >>>> have to be updated.
> >>>>
> >>>>
> >>>>
> >>>> May be we can start publishing the coverage information on wiki pages
> >>> and
> >>>> provide some updates time to time (I can do it)?
> >>>>
> >>> +1,  and of course, you can only if  no one in the mailing list objects
> >>> and you'll have my welcome, and I think it will be even greater if these
> >>> reports can be generated regularly like what JAPI is doing:)
> >>>>
> >>>> One note:
> >>>>
> >>>> I noticed that different unit tests have very different package names
> >>>>
> >>>> Now the directory with all built tests copied to one place looks like:
> >>> I think we have got a agreement on the test naming convention[1], but
> >>> for sure, there have been many (legacy) test cases before this
> >>> agreement, and I think the volunteer is highly welcome to provide patch
> >>> for them.
> >>>
> >>>
> >> [1]http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.
> >> html
> >>>>
> >>>>
> >>>> C:\coverage\tests\test>ls
> >>>>
> >>>> GZIPOutClose2.txt        api           config         javax
> >>>> tests
> >>>>
> >>>> GZIPOutFinish.txt        api.injected  dazzle         org
> >> xml
> >>>> GZIPOutWrite.txt         binary        gif            prefs
> >>>>
> >>>> Inet6Address.golden.ser  bundles       impl           serialization
> >>>>
> >>>> JDK2-3gabba.zip          com           impl.injected   test.txt
> >>>>
> >>>>
> >>>>
> >>>> I think, it would be good if tests had unified package names.
> >>>>
> >>>> Why? - so far, just common sense, just to have an order in test suite
> >>>>
> >>>> Organization (if consider all unit tests as solid test suite).
> >>>>
> >>>> Thanks,
> >>>>  Vladimir
> >>>>
> >>>> For example, my exclude list for java.io is:
> >>>> -java.io.BufferedInputStream ,
> >>>> -java.io.BufferedOutputStream,
> >>>> -java.io.File ,
> >>>> -java.io.FileChannelFactory,
> >>>> -java.io.FileDescriptor,
> >>>> -java.io.FileInputStream,
> >>>> -java.io.FileOutputStream,
> >>>> -java.io.FilterInputStream ,
> >>>> -java.io.FilterOutputStream,
> >>>> - java.io.InputStream,
> >>>> -java.io.OutputStream,
> >>>> -java.io.ObjectStreamField,
> >>>> -java.io.PrintStream
> >>>>
> >>>>
> >>>> On 6/6/06, Paulex Yang < paulex.yang@gmail.com > wrote:
> >>>>> I've attach the scripts and excluded class lists to JIRA,  please
> >> refer
> >>>>> to https://issues.apache.org/jira/browse/HARMONY-564 .  Enjoy it:).
> >>>>>
> >>>>> Mark Hindess wrote:
> >>>>>> On 2 June 2006 at 10:37, Paulex Yang < paulex.yang@gmail.com>
> >> wrote:
> >>>>>>> Mark,
> >>>>>>>
> >>>>>>> I'm glad that there is someone else has interest on emma,
I've
> >>>>> tried it
> >>>>>>> before. AFAIK, emma works by instrumentation, but sometimes
for
> >>>>> classes
> >>>>>>> in bootclasspath, the instrumentation cannot work, there
are two
> >>>>> cases:
> >>>>>
> >>>>>>> 1. Some instrumented classes cannot be loaded by VM.
> >>>>>>> 2. Some classes cannot be instrumented
> >>>>>>>
> >>>>>>> I have tried to look more inside to find some way to work
around
> >>>>> but I
> >>>>>>> haven't got enough time yet.
> >>>>>>>
> >>>>>>> Specifically for nio-channel module, I had a list for these
two
> >>> cases
> >>>>> (I
> >>>>>>> believe the data is a little outdated and should be reevaluated)
> >>>>>>> case 1.
> >>>>>>> BaseByteBuffer.class
> >>>>>>> Buffer.class
> >>>>>>> BufferFactory.class
> >>>>>>> ByteBuffer.class
> >>>>>>> CharArrayBuffer.class
> >>>>>>> CharBuffer.class
> >>>>>>> HeapByteBuffer.class
> >>>>>>> ReadWriteCharArrayBuffer.class
> >>>>>>> ReadWriteHeapByteBuffer.class
> >>>>>>> FileChannel.class
> >>>>>>> AbstractInterruptibleChannel.class
> >>>>>>> FileChannelImpl.class
> >>>>>>> WriteOnlyFileChannel.class
> >>>>>>> LockManager.class
> >>>>>>> LockManager$1.class
> >>>>>>> ReadOnlyFileChannel.class
> >>>>>>>
> >>>>>>> case 2:
> >>>>>>> ByteChannel.class
> >>>>>>> Channel.class
> >>>>>>> GatheringByteChannel.class
> >>>>>>> InterruptibleChannel.class
> >>>>>>> WritableByteChannel.class
> >>>>>>>
> >>>>>>> And I have got some ant script and more excluded list for
emma, if
> >>>>>>> anyone has interests, I can upload it to JIRA.
> >>>>>>>
> >>>>>> Yes!
> >>>>>>
> >>>>>> -Mark.
> >>>>>>
> >>>>>>
> >>>>>>> Mark Hindess wrote:
> >>>>>>>
> >>>>>>>> Anyone tried using emma (emma.sf.net) to look at test
coverage
> >>>>> for our
> >>>>>>>> tests?  That might help ensure we don't trim them down
too much?
> >>>>>>>>
> >>>>>>>> I suspect we will soon reach a point where there are
> >> significantly
> >>>>> fewer
> >>>>>>>> obvious opportunities for small contributions - e.g.
> >>>>> 1.5generification,
> >>>>>>>> etc from looking at JAPI reports.  Having test coverage
reports
> >>>>> might
> >>>>>>>> help by highlighting areas for improvement.
> >>>>>>>>
> >>>>>>>> I had a quick try with:
> >>>>>>>>
> >>>>>>>>   ant -f make/build.xml
> >>>>>>>>   ant -f modules/nio/make/build.xml test
> >>>>>>>>   java -cp emma.jar emma instr -m overwrite -cp
> >>>>> deploy/jdk/jre/lib/boot/nio
> >>>>>>> .jar
> >>>>>>>
> >>>>>>>> and then:
> >>>>>>>>
> >>>>>>>>   java -Xbootclasspath/p: emma.jar \
> >>>>>>>>        -cp build/tests:modules/nio/bin/test:junit.jar:.
\
> >>>>>>>>        junit.textui.TestRunner \
> >>>>>>>>
> >> org.apache.harmony.tests.java.nio.channels.SocketChannelTest
> >>>>>>>> (Running all the tests is a little tricky because test
depends on
> >>>>> build,
> >>>>>>>> which depends on clean which wipes out the instr'd jar.
;-( )
> >>>>>>>>
> >>>>>>>> Sadly it dies horribly with the stack trace below. 
I'll take
> >>>>> another
> >>>>>>>> look when I have a little more time, but I thought I'd
mention it
> >>> in
> >>>>>>>> case anyone else had any thoughts on the subject.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>>
> >>>>>>>> Mark.
> >>>>>>>>
> >>>>>>>> Exception in thread "main" java/lang/ExceptionInInitializerError
> >>>>>>>>         at java/lang/J9VMInternals.initialize
> >>>>> (J9VMInternals.java:203)
> >>>>>>>>         at java/nio/Buffer.$VRi ( Buffer.java:4294967295)
> >>>>>>>>         at java/nio/Buffer.<clinit> ( Buffer.java:4294967295)
> >>>>>>>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> >>>>>>>>         at java/lang/J9VMInternals.initialize (
> >> J9VMInternals.java
> >>>>> :185)
> >>>>>>>>         at java/lang/J9VMInternals.initialize
> >>>>> (J9VMInternals.java:154)
> >>>>>>>>         at java/lang/String.getBytes (String.java:783)
> >>>>>>>>         at org/apache/harmony/luni/util/Util.getBytes
(
> >>>>> Util.java:54)
> >>>>>>>>         at java/lang/ClassLoader.loadLibraryWithPath
> >>>>> (ClassLoader.java
> >>>>> :998)
> >>>>>>>>         at java/lang/ClassLoader.loadLibraryWithClassLoader
(
> >>>>> ClassLoader.ja
> >>>>>>> va:970)
> >>>>>>>
> >>>>>>>>         at java/lang/System.loadLibrary (System.java:706)
> >>>>>>>>         at com/ibm/icu4jni/common/ICU4JNILoader.loadLibrary
> >>> (Unknown
> >>>>> Source
> >>>>>>> )
> >>>>>>>
> >>>>>>>>         at com/ibm/icu4jni/converters/NativeConverter.<clinit>
> >>>>> (Unknown Sou
> >>>>>>> rce)
> >>>>>>>
> >>>>>>>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> >>>>>>>>         at java/lang/J9VMInternals.initialize
> >>>>> ( J9VMInternals.java:185)
> >>>>>>>>         at
> >>> com/ibm/icu4jni/charset/CharsetProviderICU.charsetForName
> >>>>> (Unkno
> >>>>>>> wn Source)
> >>>>>>>
> >>>>>>>>         at java/nio/charset/Charset.forNameInternal
> >>>>> (Charset.java:450)
> >>>>>>>>         at java/nio/charset/Charset.forName (Charset.java:513)
> >>>>>>>>         at java/lang/String.defaultCharset ( String.java:688)
> >>>>>>>>         at java/lang/String.<init> ( String.java:252)
> >>>>>>>>         at org/apache/harmony/luni/util/Util.toString
(Util.java
> >>> :74)
> >>>>>>>>         at java/lang/System.getPropertyList (Native
Method)
> >>>>>>>>         at java/lang/System.ensureProperties ( System.java:496)
> >>>>>>>>         at java/lang/System.<clinit> (System.java:55)
> >>>>>>>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> >>>>>>>>         at java/lang/J9VMInternals.initialize
> >>>>> (J9VMInternals.java:185)
> >>>>>>>>         at java/lang/ClassLoader.initializeClassLoaders
(
> >>>>> ClassLoader.java:6
> >>>>>>> 5)
> >>>>>>>
> >>>>>>>>         at java/lang/Thread.initialize (Thread.java:301)
> >>>>>>>>         at java/lang/Thread.<init> (Thread.java:120)
> >>>>>>>> java/lang/NullPointerException
> >>>>>>>>         at java/lang/Throwable.printStackTrace ( Throwable.java
> >>> :231)
> >>>>>>>>         at
> >> com/vladium/emma/rt/RTExitHook.createClassLoaderClosure
> >>>>> (RTExitH
> >>>>>>> ook.java:63)
> >>>>>>>
> >>>>>>>>         at com/vladium/emma/rt/RT.reset ( RT.java:92)
> >>>>>>>>         at com/vladium/emma/rt/RT.<clinit> (RT.java:238)
> >>>>>>>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> >>>>>>>>         at java/lang/J9VMInternals.initialize (
> >> J9VMInternals.java
> >>>>> :185)
> >>>>>>>>         at java/nio/Buffer.$VRi ( Buffer.java:4294967295)
> >>>>>>>>         at java/nio/Buffer.<clinit> (Buffer.java:4294967295)
> >>>>>>>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> >>>>>>>>         at java/lang/J9VMInternals.initialize
> >>>>> (J9VMInternals.java:185)
> >>>>>>>>         at java/lang/J9VMInternals.initialize
> >>>>> (J9VMInternals.java:154)
> >>>>>>>>         at java/lang/String.getBytes ( String.java :783)
> >>>>>>>>         at org/apache/harmony/luni/util/Util.getBytes
(Util.java
> >>> :54)
> >>>>>>>>         at java/lang/ClassLoader.loadLibraryWithPath
> >>>>> (ClassLoader.java
> >>>>> :998)
> >>>>>>>>         at java/lang/ClassLoader.loadLibraryWithClassLoader
(
> >>>>> ClassLoader.ja
> >>>>>>> va:970)
> >>>>>>>
> >>>>>>>>         at java/lang/System.loadLibrary ( System.java:706)
> >>>>>>>>         at com/ibm/icu4jni/common/ICU4JNILoader.loadLibrary
> >>> (Unknown
> >>>>> Source
> >>>>>>> )
> >>>>>>>
> >>>>>>>>         at com/ibm/icu4jni/converters/NativeConverter.<clinit>
> >>>>> (Unknown Sou
> >>>>>>> rce)
> >>>>>>>
> >>>>>>>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> >>>>>>>>         at java/lang/J9VMInternals.initialize
> >>>>> (J9VMInternals.java:185)
> >>>>>>>>         at
> >>> com/ibm/icu4jni/charset/CharsetProviderICU.charsetForName
> >>>>> (Unkno
> >>>>>>> wn Source)
> >>>>>>>
> >>>>>>>>         at java/nio/charset/Charset.forNameInternal
> >>>>> (Charset.java:450)
> >>>>>>>>         at java/nio/charset/Charset.forName (Charset.java:513)
> >>>>>>>>         at java/lang/String.defaultCharset ( String.java:688)
> >>>>>>>>         at java/lang/String.<init> (String.java:252)
> >>>>>>>>         at org/apache/harmony/luni/util/Util.toString
(Util.java
> >>> :74)
> >>>>>>>>         at java/lang/System.getPropertyList (Native
Method)
> >>>>>>>>         at java/lang/System.ensureProperties (System.java:496)
> >>>>>>>>         at java/lang/System.<clinit> (System.java:55)
> >>>>>>>>         at java/lang/J9VMInternals.initializeImpl (Native
Method)
> >>>>>>>>         at java/lang/J9VMInternals.initialize
> >>>>> (J9VMInternals.java:185)
> >>>>>>>>         at java/lang/ClassLoader.initializeClassLoaders
(
> >>>>> ClassLoader.java:6
> >>>>>>> 5)
> >>>>>>>
> >>>>>>>>         at java/lang/Thread.initialize (Thread.java:301)
> >>>>>>>>         at java/lang/Thread.<init> (Thread.java
:120)
> >>>>>>>> JVMJ9VM015W Initialization error for library jclclear_23(14):
> >>>>> JVMJ9VM009E J
> >>>>>>> 9VMDllMain failed
> >>>>>>>
> >>>>>>>> HMYEXEL062E Internal VM error: Failed to create Java
VM
> >>>>>>>> FAILED.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 1 June 2006 at 22:26, "Mikhail Loenko" < mloenko@gmail.com
>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> 2006/6/1, Geir Magnusson Jr < geir@pobox.com>:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> 2006/6/1, Geir Magnusson Jr < geir@pobox.com
>:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> I don't quite grok what you mean - could
you post what you
> >> are
> >>>>>>>>>>>> suggesting in code here on -dev?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> Index:
> >>>>> modules/luni/src/test/java/tests/api/java/net/Inet6AddressTest.ja
> >>>>>>> v
> >>>>>>>
> >>>>>>>>>>>
> >>>>>>>>> a
> >>>>>>>>>
> >>>>>>>>>
> >>>>> ===================================================================
> >>>>>>>>>>> ---
> >>>>>>>>>>>
> >>>>> modules/luni/src/test/java/tests/api/java/net/Inet6AddressTest.java
> >>>>>>>>>>> (revision
> >>>>>>>>>>> 410480)
> >>>>>>>>>>> +++
> >>>>>>>>>>>
> >>>>> modules/luni/src/test/java/tests/api/java/net/Inet6AddressTest.java
> >>>>>
> >>>>>>>>>>> (working
> >>>>>>>>>>> copy)
> >>>>>>>>>>> @@ -785,7 +785,7 @@
> >>>>>>>>>>>                 "FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:
> >> 255.255.255.255
> >>> ",
> >>>>>>>>>>>                 "0:0:0:0:0:0:0:0", "0:0:0:0:0:0:
0.0.0.0" };
> >>>>>>>>>>>
> >>>>>>>>>>> -        String invalidIPAddresses[] = {
":", "FFFF:FFFF",
> >>> "1:1",
> >>>>>>>>>>> "::1.2.3.444" };
> >>>>>>>>>>> +        String invalidIPAddresses[] = {
"FFFF:FFFF" };
> >>>>>>>>>>>
> >>>>>>>>>>>         for (int i = 0; i < validIPAddresses.length;
i++) {
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Thx - why do you want to remove testing the
others?  Because
> >> you
> >>>>> believe
> >>>>>>>>>> that it is in effect testing the resolver in
the OS since each
> >>>>> returns
> >>>>>>>>>> the same answer,and thus it's testing the same
thing over and
> >>>>> over?
> >>>>>>>>>>
> >>>>>>>>> I believe so, but I'd like to hear confirmation
from code
> >> authors
> >>>>>>>>> Thanks,
> >>>>>>>>> Mikhail
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>> To unsubscribe, e-mail: harmony-dev-
> >> unsubscribe@incubator.apache.org
> >>>>>>>> For additional commands, e-mail:
> >>>>> harmony-dev-help@incubator.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> --
> >>>>>>> Paulex Yang
> >>>>>>> China Software Development Lab
> >>>>>>> IBM
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>> To unsubscribe, e-mail: harmony-dev-
> >> unsubscribe@incubator.apache.org
> >>>>>>> For additional commands, e-mail:
> >>>>> harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>> -------------------------------------------------------------------
> >> --
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail: harmony-dev-
> >> unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail:
> >>> harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> --
> >>>>> Paulex Yang
> >>>>> China Software Development Lab
> >>>>> IBM
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail: harmony-dev-
> >> help@incubator.apache.org
> >>>>>
> >>>
> >>> --
> >>> Paulex Yang
> >>> China Software Development Lab
> >>> IBM
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> > 
> > 
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > 
> > 
> 
> 
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message