harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Harley <george.c.har...@googlemail.com>
Subject Re: classlib test suite status emails?
Date Tue, 11 Apr 2006 12:02:05 GMT
Stepan Mishura wrote:
> On 4/11/06, Mark Hindess wrote:
>
>   
>> Personally, obviously,  I'd expect people to run the tests before
>> committing.
>>
>> However, I notice that since enabling the security tests - which fork
>> for every test - that the tests take over half an hour to run now on
>> our Linux build machine.  So I can see why enthusiasm might lead to
>> people not running all the tests but instead perhaps just running
>> those from the module you are changing.
>>     
>
>
> I think that this message[1] relates to the problem with running big test
> suites. So should we do the similar thing i.e. define which test suites
> should be run for each module?
>
> Thanks,
> Stepan.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c442C1C79.1060805@sourcelabs.com%3e
>
>
>   

Hi Stepan,

We should both define and then automate which test suites get run for 
changes made to a given module. The dependency information is already 
there for us in the contents of each module's MANIFEST.MF file. So, to 
take a recent example which we are all familiar with ;-) , we can see 
that the java.text package exported from the text module is imported by 
modules logging, LUNI, security and SQL.

Folks wishing to verify that changes made to a given module do not break 
other modules should ideally be able to kick off a JUnit run that 
incorporates all of the tests from the other modules into one test 
suite. This could either be folded into the existing "run.tests" target 
of the module's build.xml or else be a separate target like 
"run.depends.tests" (or similar).
i.e. something like ...

<target name="run.depends.tests">

<mkdir dir="${hy.tests.reports}" />

<junit fork="yes"
forkmode="once"
printsummary="withOutAndErr"
errorproperty="test.error"
showoutput="on"
dir="${hy.text.bin.test}"
jvm="${hy.target}/jre/bin/java">

<jvmarg value="-showversion"/>

<!-- Required by various tests that set security manager etc -->
<jvmarg 
value="-Djava.security.policy=../../../../support/src/test/resources/config/testing.policy"

/>

<!-- Required for running the java.net unit tests -->
<jvmarg 
value="-Dtest.ini.file=../../../../support/src/test/resources/config/localhosttest.ini" 
/>

<env key="JAVA_HOME" value="${hy.target}/jre"/>

<!-- Adding all dependent test bins to the classpath -->
<classpath>
<pathelement path="${hy.text.bin.test}"/>
<pathelement path="${hy.logging.bin.test}"/>
<pathelement path="${hy.luni.bin.test}"/>
<pathelement path="${hy.security.bin.test}"/>
<pathelement path="${hy.sql.bin.test}"/>
<pathelement path="${hy.tests.support.bin}" />
</classpath>

<!-- The DependencyTests suite logically groups the other dependent test 
suites -->
<test name="tests.text.DependencyTests"
todir="${hy.tests.reports}"
haltonfailure="no" >
<formatter type="xml"/>
</test>
</junit>
</target>


In this scenario the class tests.text.DependencyTests would be a simple 
logical grouping of the dependent test suites using standard JUnit 
techniques.
i.e. something like ...

public class DependencyTests {
public static Test suite() {
TestSuite suite = new TestSuite("Dependents on java.text module");
suite.addTestSuite(org.apache.harmony.logging.AllTests.suite());
suite.addTestSuite(org.apache.harmony.luni.AllTests.suite());
suite.addTestSuite(org.apache.harmony.security.AllTests.suite());
suite.addTestSuite(org.apache.harmony.sql.AllTests.suite());
suite.addTestSuite(org.apache.harmony.logging.AllTests.suite());
return suite;
}
}

...and each of the AllTests classes referred to were just a logical 
grouping of the test cases in that particular module.


I think that such a scheme will be a tremendous convenience to 
developers as it will increase our confidence that breakages introduced 
into other modules by changes in a particular module will be caught soon 
without recourse to running the entire test suite - just the necessary 
ones.

If we see merit in setting up something like this to run alongside the 
existing test targets then I will be happy to open the JIRA and create 
the new Ant targets and test suites.

Best regards,
George

>   
>> George just commented in a commit message backing out some changes in
>> text that all the text tests passed even though the changed caused
>> failures elsewhere.
>>
>> Regards,
>> Mark.
>>
>> On 4/11/06, Geir Magnusson Jr <geir@pobox.com> wrote:
>>     
>>> Just curious (and this isn't a criticism - I'm just as guilty of not
>>> doing this)...
>>>
>>> Don't you run the tests before committing?
>>>
>>> geir
>>>
>>> George Harley wrote:
>>>       
>>>> Hi,
>>>>
>>>> It *seems* like things started failing after I committed the changes
>>>>         
>> for
>>     
>>>> HARMONY-205 last night. I'm looking into this now. If the
>>>>         
>> investigation
>>     
>>>> begins to take up too much time I will back the changes out.
>>>>
>>>> Best regards,
>>>> George
>>>>
>>>>
>>>> Stepan Mishura wrote:
>>>>         
>>>>> The same for me + DatagramChannelTest
>>>>>
>>>>> Thanks,
>>>>> Stepan.
>>>>>
>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>
>>>>>           
>>>>>> No.  These:
>>>>>>
>>>>>> F
>>>>>>
>>>>>>             
>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoder
>>     
>>>>>> E
>>>>>>
>>>>>>             
>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder01
>>     
>>>>>> E
>>>>>>
>>>>>>             
>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder02
>>     
>>>>>> F org.apache.harmony.security.asn1.der.DerUTCTimeEDTest.testMt
>>>>>> E
>>>>>>
>>>>>>             
>> org.apache.harmony.security.x509.PrivateKeyUsagePeriodTest.testEncodeDecode
>>     
>>>>>> F java.security.cert.X509CertSelectorTest.testSetPrivateKeyValid
>>>>>> F java.security.cert.X509CertSelectorTest.testMatch
>>>>>> F java.security.cert.X509CertSelectorTest.testClone
>>>>>> F tests.api.java.sql.DateTest.testSetTimelong
>>>>>> F tests.api.java.sql.DateTest.testToString
>>>>>> F tests.api.java.sql.DateTest.testValueOf
>>>>>> F tests.api.java.sql.TimestampTest.testSetNanosint
>>>>>> F tests.api.java.sql.TimestampTest.testToString
>>>>>> F tests.api.java.util.DateTest.test_toGMTString
>>>>>> F tests.api.java.util.DateTest.test_toString
>>>>>>
>>>>>> Regards,
>>>>>> Mark.
>>>>>>
>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com> wrote:
>>>>>>
>>>>>>             
>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>
>>>>>>>               
>>>>>>>> Yes.  I was using the failureproperty mechanism.  Trying
to get
>>>>>>>>                 
>> this
>>     
>>>>>>>> property propogated back to the top level ant file was what
I was
>>>>>>>> having trouble with.
>>>>>>>>
>>>>>>>> Using a file as you suggest might help.  I'll give that a
try
>>>>>>>>
>>>>>>>>                 
>>>>>> shortly...
>>>>>>
>>>>>>             
>>>>>>>> Incidentally, I'm seeing 12 failures and 3 errors on r393111.
>>>>>>>>
>>>>>>>>                 
>>>>>>>  I guess that you have 9 tests from DatagramChannelTest passed.
And
>>>>>>> 12 +
>>>>>>>
>>>>>>>               
>>>>>> 3 =
>>>>>>
>>>>>>             
>>>>>>> 15 :-)
>>>>>>>
>>>>>>>  (And
>>>>>>>
>>>>>>>               
>>>>>>>> there are typos - "mathc" should be "match" - in the failure
>>>>>>>>                 
>> messages
>>     
>>>>>>>> for java.security.cert testMatch and testClone.)
>>>>>>>>
>>>>>>>>                 
>>>>>>> I've fixed typo.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Stepan.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>               
>>>>>>>> Mark.
>>>>>>>>
>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> Stepan,
>>>>>>>>>>
>>>>>>>>>> I have another build running (but without notifications
going to
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>> the
>>>>>>
>>>>>>             
>>>>>>>>>> list) that runs:
>>>>>>>>>>
>>>>>>>>>> 1) build (with reference jdk)
>>>>>>>>>> 2) build with what we created with 1)
>>>>>>>>>> 3) test
>>>>>>>>>> 4) create classlists and compare with class load
data for
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>> applications
>>>>>>
>>>>>>             
>>>>>>>>>>    (tomcat, geronimo, continuum, etc.)
>>>>>>>>>> 5) generate JAPI results
>>>>>>>>>>
>>>>>>>>>> I'd like to fail this build at step 3, but I can't
see an easy
>>>>>>>>>>                     
>> way
>>     
>>>>>> to
>>>>>>
>>>>>>             
>>>>>>>>>> get 'ant -f make/build.xml test' to run all tests
and then fail
>>>>>>>>>>                     
>> if
>>     
>>>>>> any
>>>>>>
>>>>>>             
>>>>>>>>>> of the module test sub-targets had test failures.
 I could parse
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>> the
>>>>>>
>>>>>>             
>>>>>>>>>> output I suppose, but I'd rather get ant to propagate
the junit
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>> tasks
>>>>>>
>>>>>>             
>>>>>>>>>> failure property back up to the top level.  I've
tried a couple
>>>>>>>>>>                     
>> of
>>     
>>>>>>>>>> things and none seem to work.  Any suggestions welcome.
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>  Mark, did you try failureproperty parameter for junit
task?
>>>>>>>>> We may add it to ant sub-targets to raise a flag, for
example,
>>>>>>>>>
>>>>>>>>>                   
>>>>>> create
>>>>>>
>>>>>>             
>>>>>>>> file "
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> TESTS.FAILED" in the root,  when tests for some module
fail. And
>>>>>>>>>                   
>> in
>>     
>>>>>> the
>>>>>>
>>>>>>             
>>>>>>>> end
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> of tests suite run check whether this file exists on
not.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Stepan.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> Mark.
>>>>>>>>>>
>>>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I've checked out at morning last updates, built
the code base
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>> and
>>>>>>
>>>>>>             
>>>>>>>> run
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> tests …and there are 24 tests failures!
>>>>>>>>>>>
>>>>>>>>>>> There are 9 tests failures in
>>>>>>>>>>> org.apache.harmony.tests.java.nio.channels.DatagramChannelTest–
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>> I
>>>>>>
>>>>>>             
>>>>>>>> saw
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> these
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> failures before from time to time. It seems that
tests depend
>>>>>>>>>>>                       
>> on
>>     
>>>>>>>> some
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> race
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> conditions because they may pass if I rerun them.
Paulex, are
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>> these
>>>>>>
>>>>>>             
>>>>>>>>>> tests
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> passing for you?
>>>>>>>>>>>
>>>>>>>>>>> And there are new 15 test failures.  So now if
I'll make a code
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>> update
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> or a
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> bug fix how I can be 100% sure that I don't do
any regression?
>>>>>>>>>>>
>>>>>>>>>>> Currently if a commit breaks the Harmony classlib
build a
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>> notification
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> with
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> subject: "[continuum] BUILD FAILURE: Classlib/linux.ia32"
will
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>> be
>>>>>>
>>>>>>             
>>>>>>>> send
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> harmony-commits mailing list. Is it possible
to have the same
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>> for
>>>>>>
>>>>>>             
>>>>>>>> tests?
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> I
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> mean that after completing automatic build the
existing
>>>>>>>>>>>                       
>> classlib
>>     
>>>>>>>> tests
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> suite
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> should be run. If there are failing tests then
a report
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>> notification
>>>>>>
>>>>>>             
>>>>>>>>>> with
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> corresponding subject will be send.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Stepan Mishura
>>>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>> --
>>>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>> ---------------------------------------------------------------------
>>     
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>
>>>>>>             
>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>             
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>> ---------------------------------------------------------------------
>>     
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>                   
>> harmony-dev-unsubscribe@incubator.apache.org
>>     
>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>
>>>>>>>>>                   
>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>             
>>>>>>>>> Thanks,
>>>>>>>>> Stepan Mishura
>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> --
>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>> ---------------------------------------------------------------------
>>     
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>                 
>> harmony-dev-unsubscribe@incubator.apache.org
>>     
>>>>>>>> For additional commands, e-mail:
>>>>>>>>                 
>> harmony-dev-help@incubator.apache.org
>>     
>>>>>>>>
>>>>>>>>                 
>>>>>>> --
>>>>>>>
>>>>>>>               
>> ---------------------------------------------------------------------
>>     
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail:
>>>>>>>               
>> harmony-dev-unsubscribe@incubator.apache.org
>>     
>>>>>>> For additional commands, e-mail:
>>>>>>>               
>> harmony-dev-help@incubator.apache.org
>>     
>>>>>>> Thanks,
>>>>>>> Stepan Mishura
>>>>>>> Intel Middleware Products Division
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> --
>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>> IBM Java Technology Centre, UK.
>>>>>>
>>>>>>
>>>>>>             
>> ---------------------------------------------------------------------
>>     
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail:
>>>>>>             
>> harmony-dev-help@incubator.apache.org
>>     
>>>>>>
>>>>>>             
>>>>> --
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail:
>>>>>           
>> harmony-dev-help@incubator.apache.org
>>     
>>>>> Thanks,
>>>>> Stepan Mishura
>>>>> Intel Middleware Products Division
>>>>>
>>>>>
>>>>>           
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>         
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>       
>> --
>> Mark Hindess <mark.hindess@googlemail.com>
>> IBM Java Technology Centre, UK.
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>     
>
>
> --
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
> Thanks,
> Stepan Mishura
> Intel Middleware Products Division
>
>   


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message