harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject Re: classlib test suite status emails?
Date Tue, 11 Apr 2006 14:46:48 GMT


George Harley wrote:
> Geir Magnusson Jr wrote:
>>
>>
>> George Harley wrote:
>>> Hi Geir,
>>>
>>> From my point of view the answer is yes, I ran the tests for the 
>>> patched module. The build/test server picked up the downstream 
>>> breakages in the other modules. I fixed matters so that the builds 
>>> ran again.
>>
>> Right - I thought we considered the build/test server a safety to keep 
>> accidents from getting away from us, but given the delay and the 
>> distribution of people throughout the planet working on this (rather 
>> than all sitting in the same office), we want to avoid breakages at 
>> all costs. (Again, I'm guilty of the same thing...)
>>
>>>
>>> Elsewhere in this thread I am trying to cooperate with others to make 
>>> it simpler to run more - just the necessary - tests prior to making a 
>>> commit.
>>
>> I guess the question is "what is necessary"? Given a dependency chain
>>
>> A <- B <- C
>>
>> I don't think it's only the immediate dependencies of a given module 
>> A, but rather the full dependency graph, because a "1 away" module (C) 
>> might use a direct dependency (B) in a way different than the test 
>> cases for that direct dependency do (especially since we aren't quite 
>> up to speed with doing implementation testing...)
> 
> Hi Geir,
> 
> Sure, point taken. Testing A and then its set of immediate dependants 
> (just B in this example) will greatly reduce the risk of downstream 
> module breakages but will not completely catch all such events. 
> Actually, subject to the quality of the tests in C, it is possible that 
> even testing A, B and C will not find any such submerged problems.

Indeed.

> 
> I am not an expert in the theory of testing ( as if you had to be told 
> :-) ) but instinct alone tells me that making it convenient and quick to 
> run a module's tests and those of its dependants will definitely 
> *improve* matters - it would certainly have helped me last night before 
> I broke the build.

There is a cultural element to it - maybe we keep tabs on who breaks the 
build, and that decides who buys the beer next time we all meet...

geir

> 
> Best regards,
> George
> 
>>
>>>
>>> Thank you for your support.
>>>
>>
>> Um...
>> geir
>>
>>>
>>> Best regards,
>>> George
>>>
>>>
>>> Geir Magnusson Jr wrote:
>>>> So the answer is "no"?
>>>>
>>>> George Harley wrote:
>>>>> Geir Magnusson Jr wrote:
>>>>>> Just curious (and this isn't a criticism - I'm just as guilty of

>>>>>> not doing this)...
>>>>>>
>>>>>> Don't you run the tests before committing?
>>>>>
>>>>> Hi Geir,
>>>>>
>>>>> Depends what do you mean by "the tests".
>>>>>
>>>>> The change was completely encapsulated in the text component. I ran 
>>>>> the text tests. I checked that the new tests introduced in the 
>>>>> issue patch gave the same results when run against a 5.0 RI. All 
>>>>> was fine.
>>>>>
>>>>> The automated build and test server (which runs all the tests in 
>>>>> all the modules) found the problem. That is it's job and it did it 
>>>>> well. The problem was found relatively quickly and has now been 
>>>>> remedied. The turnaround time would have been quicker if the 
>>>>> automated build and test server issued a failure notice on failing 
>>>>> tests but that is now in the process of getting sorted out.
>>>>>
>>>>> IMHO this problem was a bit of an odd problem as the failures did 
>>>>> not arise in the patched module's test suite but in other modules. 
>>>>> That is a good indication that the test suite for text is in need 
>>>>> of some scrutiny. That is a useful thing to be made aware of.
>>>>>
>>>>>
>>>>> Best regards,
>>>>> George
>>>>>
>>>>>>
>>>>>> geir
>>>>>>
>>>>>> George Harley wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> It *seems* like things started failing after I committed the

>>>>>>> changes for HARMONY-205 last night. I'm looking into this now.
If 
>>>>>>> the investigation begins to take up too much time I will back
the 
>>>>>>> changes out.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> George
>>>>>>>
>>>>>>>
>>>>>>> Stepan Mishura wrote:
>>>>>>>> The same for me + DatagramChannelTest
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Stepan.
>>>>>>>>
>>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>>
>>>>>>>>> No. These:
>>>>>>>>>
>>>>>>>>> F
>>>>>>>>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoder

>>>>>>>>>
>>>>>>>>> E
>>>>>>>>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder01

>>>>>>>>>
>>>>>>>>> E
>>>>>>>>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder02

>>>>>>>>>
>>>>>>>>> F org.apache.harmony.security.asn1.der.DerUTCTimeEDTest.testMt
>>>>>>>>> E
>>>>>>>>> org.apache.harmony.security.x509.PrivateKeyUsagePeriodTest.testEncodeDecode

>>>>>>>>>
>>>>>>>>> F java.security.cert.X509CertSelectorTest.testSetPrivateKeyValid
>>>>>>>>> F java.security.cert.X509CertSelectorTest.testMatch
>>>>>>>>> F java.security.cert.X509CertSelectorTest.testClone
>>>>>>>>> F tests.api.java.sql.DateTest.testSetTimelong
>>>>>>>>> F tests.api.java.sql.DateTest.testToString
>>>>>>>>> F tests.api.java.sql.DateTest.testValueOf
>>>>>>>>> F tests.api.java.sql.TimestampTest.testSetNanosint
>>>>>>>>> F tests.api.java.sql.TimestampTest.testToString
>>>>>>>>> F tests.api.java.util.DateTest.test_toGMTString
>>>>>>>>> F tests.api.java.util.DateTest.test_toString
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Mark.
>>>>>>>>>
>>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>>>>> Yes. I was using the failureproperty mechanism.
Trying to get 
>>>>>>>>>>> this
>>>>>>>>>>> property propogated back to the top level ant
file was what I 
>>>>>>>>>>> was
>>>>>>>>>>> having trouble with.
>>>>>>>>>>>
>>>>>>>>>>> Using a file as you suggest might help. I'll
give that a try
>>>>>>>>> shortly...
>>>>>>>>>>> Incidentally, I'm seeing 12 failures and 3 errors
on r393111.
>>>>>>>>>> I guess that you have 9 tests from DatagramChannelTest
passed. 
>>>>>>>>>> And 12 +
>>>>>>>>> 3 =
>>>>>>>>>> 15 :-)
>>>>>>>>>>
>>>>>>>>>> (And
>>>>>>>>>>> there are typos - "mathc" should be "match" -
in the failure 
>>>>>>>>>>> messages
>>>>>>>>>>> for java.security.cert testMatch and testClone.)
>>>>>>>>>> I've fixed typo.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Stepan.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>> Mark.
>>>>>>>>>>>
>>>>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Stepan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have another build running (but without
notifications 
>>>>>>>>>>>>> going to
>>>>>>>>> the
>>>>>>>>>>>>> list) that runs:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) build (with reference jdk)
>>>>>>>>>>>>> 2) build with what we created with 1)
>>>>>>>>>>>>> 3) test
>>>>>>>>>>>>> 4) create classlists and compare with
class load data for
>>>>>>>>> applications
>>>>>>>>>>>>> (tomcat, geronimo, continuum, etc.)
>>>>>>>>>>>>> 5) generate JAPI results
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd like to fail this build at step 3,
but I can't see an 
>>>>>>>>>>>>> easy way
>>>>>>>>> to
>>>>>>>>>>>>> get 'ant -f make/build.xml test' to run
all tests and then 
>>>>>>>>>>>>> fail if
>>>>>>>>> any
>>>>>>>>>>>>> of the module test sub-targets had test
failures. I could 
>>>>>>>>>>>>> parse
>>>>>>>>> the
>>>>>>>>>>>>> output I suppose, but I'd rather get
ant to propagate the 
>>>>>>>>>>>>> junit
>>>>>>>>> tasks
>>>>>>>>>>>>> failure property back up to the top level.
I've tried a 
>>>>>>>>>>>>> couple of
>>>>>>>>>>>>> things and none seem to work. Any suggestions
welcome.
>>>>>>>>>>>> Mark, did you try failureproperty parameter
for junit task?
>>>>>>>>>>>> We may add it to ant sub-targets to raise
a flag, for example,
>>>>>>>>> create
>>>>>>>>>>> file "
>>>>>>>>>>>> TESTS.FAILED" in the root, when tests for
some module fail. 
>>>>>>>>>>>> And in
>>>>>>>>> the
>>>>>>>>>>> end
>>>>>>>>>>>> of tests suite run check whether this file
exists on not.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Stepan.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Mark.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've checked out at morning last
updates, built the code base
>>>>>>>>> and
>>>>>>>>>>> run
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> tests …and there are 24 tests failures!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There are 9 tests failures in
>>>>>>>>>>>>>> org.apache.harmony.tests.java.nio.channels.DatagramChannelTest

>>>>>>>>>>>>>> –
>>>>>>>>> I
>>>>>>>>>>> saw
>>>>>>>>>>>>> these
>>>>>>>>>>>>>> failures before from time to time.
It seems that tests 
>>>>>>>>>>>>>> depend on
>>>>>>>>>>> some
>>>>>>>>>>>>> race
>>>>>>>>>>>>>> conditions because they may pass
if I rerun them. Paulex, are
>>>>>>>>> these
>>>>>>>>>>>>> tests
>>>>>>>>>>>>>> passing for you?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And there are new 15 test failures.
So now if I'll make a 
>>>>>>>>>>>>>> code
>>>>>>>>>>> update
>>>>>>>>>>>>> or a
>>>>>>>>>>>>>> bug fix how I can be 100% sure that
I don't do any 
>>>>>>>>>>>>>> regression?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Currently if a commit breaks the
Harmony classlib build a
>>>>>>>>>>> notification
>>>>>>>>>>>>> with
>>>>>>>>>>>>>> subject: "[continuum] BUILD FAILURE:
Classlib/linux.ia32" 
>>>>>>>>>>>>>> will
>>>>>>>>> be
>>>>>>>>>>> send
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> harmony-commits mailing list. Is
it possible to have the same
>>>>>>>>> for
>>>>>>>>>>> tests?
>>>>>>>>>>>>> I
>>>>>>>>>>>>>> mean that after completing automatic
build the existing 
>>>>>>>>>>>>>> classlib
>>>>>>>>>>> tests
>>>>>>>>>>>>> suite
>>>>>>>>>>>>>> should be run. If there are failing
tests then a report
>>>>>>>>> notification
>>>>>>>>>>>>> with
>>>>>>>>>>>>>> corresponding subject will be send.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Stepan Mishura
>>>>>>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>
>>>>>>>>>>>>> Terms of use : 
>>>>>>>>>>>>> http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>>>>> -- 
>>>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>
>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Stepan Mishura
>>>>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>>>
>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>> For additional commands, e-mail: 
>>>>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>>
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>> For additional commands, e-mail: 
>>>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Stepan Mishura
>>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: 
>>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: 
>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Stepan Mishura
>>>>>>>> Intel Middleware Products Division
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------

>>>>>>>
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: 
>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: 
>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
> 
> 
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> 
> 

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message