harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject Re: classlib test suite status emails?
Date Tue, 11 Apr 2006 12:02:54 GMT


Stepan Mishura wrote:
> On 4/11/06, Mark Hindess wrote:
> 
>> Personally, obviously,  I'd expect people to run the tests before
>> committing.
>>
>> However, I notice that since enabling the security tests - which fork
>> for every test - that the tests take over half an hour to run now on
>> our Linux build machine.  So I can see why enthusiasm might lead to
>> people not running all the tests but instead perhaps just running
>> those from the module you are changing.
> 
> 
> I think that this message[1] relates to the problem with running big test
> suites. So should we do the similar thing i.e. define which test suites
> should be run for each module?

The problem is side effects (as George discovered...)

geir

> 
> Thanks,
> Stepan.
> 
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/%3c442C1C79.1060805@sourcelabs.com%3e
> 
> 
> 
>> George just commented in a commit message backing out some changes in
>> text that all the text tests passed even though the changed caused
>> failures elsewhere.
>>
>> Regards,
>> Mark.
>>
>> On 4/11/06, Geir Magnusson Jr <geir@pobox.com> wrote:
>>> Just curious (and this isn't a criticism - I'm just as guilty of not
>>> doing this)...
>>>
>>> Don't you run the tests before committing?
>>>
>>> geir
>>>
>>> George Harley wrote:
>>>> Hi,
>>>>
>>>> It *seems* like things started failing after I committed the changes
>> for
>>>> HARMONY-205 last night. I'm looking into this now. If the
>> investigation
>>>> begins to take up too much time I will back the changes out.
>>>>
>>>> Best regards,
>>>> George
>>>>
>>>>
>>>> Stepan Mishura wrote:
>>>>> The same for me + DatagramChannelTest
>>>>>
>>>>> Thanks,
>>>>> Stepan.
>>>>>
>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>
>>>>>> No.  These:
>>>>>>
>>>>>> F
>>>>>>
>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoder
>>>>>> E
>>>>>>
>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder01
>>>>>> E
>>>>>>
>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder02
>>>>>> F org.apache.harmony.security.asn1.der.DerUTCTimeEDTest.testMt
>>>>>> E
>>>>>>
>> org.apache.harmony.security.x509.PrivateKeyUsagePeriodTest.testEncodeDecode
>>>>>> F java.security.cert.X509CertSelectorTest.testSetPrivateKeyValid
>>>>>> F java.security.cert.X509CertSelectorTest.testMatch
>>>>>> F java.security.cert.X509CertSelectorTest.testClone
>>>>>> F tests.api.java.sql.DateTest.testSetTimelong
>>>>>> F tests.api.java.sql.DateTest.testToString
>>>>>> F tests.api.java.sql.DateTest.testValueOf
>>>>>> F tests.api.java.sql.TimestampTest.testSetNanosint
>>>>>> F tests.api.java.sql.TimestampTest.testToString
>>>>>> F tests.api.java.util.DateTest.test_toGMTString
>>>>>> F tests.api.java.util.DateTest.test_toString
>>>>>>
>>>>>> Regards,
>>>>>> Mark.
>>>>>>
>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com> wrote:
>>>>>>
>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>
>>>>>>>> Yes.  I was using the failureproperty mechanism.  Trying
to get
>> this
>>>>>>>> property propogated back to the top level ant file was what
I was
>>>>>>>> having trouble with.
>>>>>>>>
>>>>>>>> Using a file as you suggest might help.  I'll give that a
try
>>>>>>>>
>>>>>> shortly...
>>>>>>
>>>>>>>> Incidentally, I'm seeing 12 failures and 3 errors on r393111.
>>>>>>>>
>>>>>>>  I guess that you have 9 tests from DatagramChannelTest passed.
And
>>>>>>> 12 +
>>>>>>>
>>>>>> 3 =
>>>>>>
>>>>>>> 15 :-)
>>>>>>>
>>>>>>>  (And
>>>>>>>
>>>>>>>> there are typos - "mathc" should be "match" - in the failure
>> messages
>>>>>>>> for java.security.cert testMatch and testClone.)
>>>>>>>>
>>>>>>> I've fixed typo.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Stepan.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>> Mark.
>>>>>>>>
>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>
>>>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Stepan,
>>>>>>>>>>
>>>>>>>>>> I have another build running (but without notifications
going to
>>>>>>>>>>
>>>>>> the
>>>>>>
>>>>>>>>>> list) that runs:
>>>>>>>>>>
>>>>>>>>>> 1) build (with reference jdk)
>>>>>>>>>> 2) build with what we created with 1)
>>>>>>>>>> 3) test
>>>>>>>>>> 4) create classlists and compare with class load
data for
>>>>>>>>>>
>>>>>> applications
>>>>>>
>>>>>>>>>>    (tomcat, geronimo, continuum, etc.)
>>>>>>>>>> 5) generate JAPI results
>>>>>>>>>>
>>>>>>>>>> I'd like to fail this build at step 3, but I can't
see an easy
>> way
>>>>>> to
>>>>>>
>>>>>>>>>> get 'ant -f make/build.xml test' to run all tests
and then fail
>> if
>>>>>> any
>>>>>>
>>>>>>>>>> of the module test sub-targets had test failures.
 I could parse
>>>>>>>>>>
>>>>>> the
>>>>>>
>>>>>>>>>> output I suppose, but I'd rather get ant to propagate
the junit
>>>>>>>>>>
>>>>>> tasks
>>>>>>
>>>>>>>>>> failure property back up to the top level.  I've
tried a couple
>> of
>>>>>>>>>> things and none seem to work.  Any suggestions welcome.
>>>>>>>>>>
>>>>>>>>>  Mark, did you try failureproperty parameter for junit
task?
>>>>>>>>> We may add it to ant sub-targets to raise a flag, for
example,
>>>>>>>>>
>>>>>> create
>>>>>>
>>>>>>>> file "
>>>>>>>>
>>>>>>>>> TESTS.FAILED" in the root,  when tests for some module
fail. And
>> in
>>>>>> the
>>>>>>
>>>>>>>> end
>>>>>>>>
>>>>>>>>> of tests suite run check whether this file exists on
not.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Stepan.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>>> Mark.
>>>>>>>>>>
>>>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I've checked out at morning last updates, built
the code base
>>>>>>>>>>>
>>>>>> and
>>>>>>
>>>>>>>> run
>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> tests …and there are 24 tests failures!
>>>>>>>>>>>
>>>>>>>>>>> There are 9 tests failures in
>>>>>>>>>>> org.apache.harmony.tests.java.nio.channels.DatagramChannelTest–
>>>>>>>>>>>
>>>>>> I
>>>>>>
>>>>>>>> saw
>>>>>>>>
>>>>>>>>>> these
>>>>>>>>>>
>>>>>>>>>>> failures before from time to time. It seems that
tests depend
>> on
>>>>>>>> some
>>>>>>>>
>>>>>>>>>> race
>>>>>>>>>>
>>>>>>>>>>> conditions because they may pass if I rerun them.
Paulex, are
>>>>>>>>>>>
>>>>>> these
>>>>>>
>>>>>>>>>> tests
>>>>>>>>>>
>>>>>>>>>>> passing for you?
>>>>>>>>>>>
>>>>>>>>>>> And there are new 15 test failures.  So now if
I'll make a code
>>>>>>>>>>>
>>>>>>>> update
>>>>>>>>
>>>>>>>>>> or a
>>>>>>>>>>
>>>>>>>>>>> bug fix how I can be 100% sure that I don't do
any regression?
>>>>>>>>>>>
>>>>>>>>>>> Currently if a commit breaks the Harmony classlib
build a
>>>>>>>>>>>
>>>>>>>> notification
>>>>>>>>
>>>>>>>>>> with
>>>>>>>>>>
>>>>>>>>>>> subject: "[continuum] BUILD FAILURE: Classlib/linux.ia32"
will
>>>>>>>>>>>
>>>>>> be
>>>>>>
>>>>>>>> send
>>>>>>>>
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>> harmony-commits mailing list. Is it possible
to have the same
>>>>>>>>>>>
>>>>>> for
>>>>>>
>>>>>>>> tests?
>>>>>>>>
>>>>>>>>>> I
>>>>>>>>>>
>>>>>>>>>>> mean that after completing automatic build the
existing
>> classlib
>>>>>>>> tests
>>>>>>>>
>>>>>>>>>> suite
>>>>>>>>>>
>>>>>>>>>>> should be run. If there are failing tests then
a report
>>>>>>>>>>>
>>>>>> notification
>>>>>>
>>>>>>>>>> with
>>>>>>>>>>
>>>>>>>>>>> corresponding subject will be send.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Stepan Mishura
>>>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>
>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>
>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>
>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail:
>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>
>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Stepan Mishura
>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail:
>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail:
>> harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail:
>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail:
>> harmony-dev-help@incubator.apache.org
>>>>>>> Thanks,
>>>>>>> Stepan Mishura
>>>>>>> Intel Middleware Products Division
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>> IBM Java Technology Centre, UK.
>>>>>>
>>>>>>
>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail:
>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail:
>> harmony-dev-help@incubator.apache.org
>>>>> Thanks,
>>>>> Stepan Mishura
>>>>> Intel Middleware Products Division
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>
>> --
>> Mark Hindess <mark.hindess@googlemail.com>
>> IBM Java Technology Centre, UK.
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
> 
> 
> --
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> 
> Thanks,
> Stepan Mishura
> Intel Middleware Products Division
> 

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message