harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Harley <george.c.har...@googlemail.com>
Subject Re: classlib test suite status emails?
Date Tue, 11 Apr 2006 14:42:35 GMT
Geir Magnusson Jr wrote:
>
>
> George Harley wrote:
>> Hi Geir,
>>
>> From my point of view the answer is yes, I ran the tests for the 
>> patched module. The build/test server picked up the downstream 
>> breakages in the other modules. I fixed matters so that the builds 
>> ran again.
>
> Right - I thought we considered the build/test server a safety to keep 
> accidents from getting away from us, but given the delay and the 
> distribution of people throughout the planet working on this (rather 
> than all sitting in the same office), we want to avoid breakages at 
> all costs. (Again, I'm guilty of the same thing...)
>
>>
>> Elsewhere in this thread I am trying to cooperate with others to make 
>> it simpler to run more - just the necessary - tests prior to making a 
>> commit.
>
> I guess the question is "what is necessary"? Given a dependency chain
>
> A <- B <- C
>
> I don't think it's only the immediate dependencies of a given module 
> A, but rather the full dependency graph, because a "1 away" module (C) 
> might use a direct dependency (B) in a way different than the test 
> cases for that direct dependency do (especially since we aren't quite 
> up to speed with doing implementation testing...)

Hi Geir,

Sure, point taken. Testing A and then its set of immediate dependants 
(just B in this example) will greatly reduce the risk of downstream 
module breakages but will not completely catch all such events. 
Actually, subject to the quality of the tests in C, it is possible that 
even testing A, B and C will not find any such submerged problems.

I am not an expert in the theory of testing ( as if you had to be told 
:-) ) but instinct alone tells me that making it convenient and quick to 
run a module's tests and those of its dependants will definitely 
*improve* matters - it would certainly have helped me last night before 
I broke the build.

Best regards,
George

>
>>
>> Thank you for your support.
>>
>
> Um...
> geir
>
>>
>> Best regards,
>> George
>>
>>
>> Geir Magnusson Jr wrote:
>>> So the answer is "no"?
>>>
>>> George Harley wrote:
>>>> Geir Magnusson Jr wrote:
>>>>> Just curious (and this isn't a criticism - I'm just as guilty of 
>>>>> not doing this)...
>>>>>
>>>>> Don't you run the tests before committing?
>>>>
>>>> Hi Geir,
>>>>
>>>> Depends what do you mean by "the tests".
>>>>
>>>> The change was completely encapsulated in the text component. I ran 
>>>> the text tests. I checked that the new tests introduced in the 
>>>> issue patch gave the same results when run against a 5.0 RI. All 
>>>> was fine.
>>>>
>>>> The automated build and test server (which runs all the tests in 
>>>> all the modules) found the problem. That is it's job and it did it 
>>>> well. The problem was found relatively quickly and has now been 
>>>> remedied. The turnaround time would have been quicker if the 
>>>> automated build and test server issued a failure notice on failing 
>>>> tests but that is now in the process of getting sorted out.
>>>>
>>>> IMHO this problem was a bit of an odd problem as the failures did 
>>>> not arise in the patched module's test suite but in other modules. 
>>>> That is a good indication that the test suite for text is in need 
>>>> of some scrutiny. That is a useful thing to be made aware of.
>>>>
>>>>
>>>> Best regards,
>>>> George
>>>>
>>>>>
>>>>> geir
>>>>>
>>>>> George Harley wrote:
>>>>>> Hi,
>>>>>>
>>>>>> It *seems* like things started failing after I committed the 
>>>>>> changes for HARMONY-205 last night. I'm looking into this now. If

>>>>>> the investigation begins to take up too much time I will back the

>>>>>> changes out.
>>>>>>
>>>>>> Best regards,
>>>>>> George
>>>>>>
>>>>>>
>>>>>> Stepan Mishura wrote:
>>>>>>> The same for me + DatagramChannelTest
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Stepan.
>>>>>>>
>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>
>>>>>>>> No. These:
>>>>>>>>
>>>>>>>> F
>>>>>>>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoder

>>>>>>>>
>>>>>>>> E
>>>>>>>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder01

>>>>>>>>
>>>>>>>> E
>>>>>>>> org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder02

>>>>>>>>
>>>>>>>> F org.apache.harmony.security.asn1.der.DerUTCTimeEDTest.testMt
>>>>>>>> E
>>>>>>>> org.apache.harmony.security.x509.PrivateKeyUsagePeriodTest.testEncodeDecode

>>>>>>>>
>>>>>>>> F java.security.cert.X509CertSelectorTest.testSetPrivateKeyValid
>>>>>>>> F java.security.cert.X509CertSelectorTest.testMatch
>>>>>>>> F java.security.cert.X509CertSelectorTest.testClone
>>>>>>>> F tests.api.java.sql.DateTest.testSetTimelong
>>>>>>>> F tests.api.java.sql.DateTest.testToString
>>>>>>>> F tests.api.java.sql.DateTest.testValueOf
>>>>>>>> F tests.api.java.sql.TimestampTest.testSetNanosint
>>>>>>>> F tests.api.java.sql.TimestampTest.testToString
>>>>>>>> F tests.api.java.util.DateTest.test_toGMTString
>>>>>>>> F tests.api.java.util.DateTest.test_toString
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Mark.
>>>>>>>>
>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>>>> Yes. I was using the failureproperty mechanism. Trying
to get 
>>>>>>>>>> this
>>>>>>>>>> property propogated back to the top level ant file
was what I 
>>>>>>>>>> was
>>>>>>>>>> having trouble with.
>>>>>>>>>>
>>>>>>>>>> Using a file as you suggest might help. I'll give
that a try
>>>>>>>> shortly...
>>>>>>>>>> Incidentally, I'm seeing 12 failures and 3 errors
on r393111.
>>>>>>>>> I guess that you have 9 tests from DatagramChannelTest
passed. 
>>>>>>>>> And 12 +
>>>>>>>> 3 =
>>>>>>>>> 15 :-)
>>>>>>>>>
>>>>>>>>> (And
>>>>>>>>>> there are typos - "mathc" should be "match" - in
the failure 
>>>>>>>>>> messages
>>>>>>>>>> for java.security.cert testMatch and testClone.)
>>>>>>>>> I've fixed typo.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Stepan.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>> Mark.
>>>>>>>>>>
>>>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>>>> On 4/11/06, Mark Hindess wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Stepan,
>>>>>>>>>>>>
>>>>>>>>>>>> I have another build running (but without
notifications 
>>>>>>>>>>>> going to
>>>>>>>> the
>>>>>>>>>>>> list) that runs:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) build (with reference jdk)
>>>>>>>>>>>> 2) build with what we created with 1)
>>>>>>>>>>>> 3) test
>>>>>>>>>>>> 4) create classlists and compare with class
load data for
>>>>>>>> applications
>>>>>>>>>>>> (tomcat, geronimo, continuum, etc.)
>>>>>>>>>>>> 5) generate JAPI results
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to fail this build at step 3, but
I can't see an 
>>>>>>>>>>>> easy way
>>>>>>>> to
>>>>>>>>>>>> get 'ant -f make/build.xml test' to run all
tests and then 
>>>>>>>>>>>> fail if
>>>>>>>> any
>>>>>>>>>>>> of the module test sub-targets had test failures.
I could 
>>>>>>>>>>>> parse
>>>>>>>> the
>>>>>>>>>>>> output I suppose, but I'd rather get ant
to propagate the 
>>>>>>>>>>>> junit
>>>>>>>> tasks
>>>>>>>>>>>> failure property back up to the top level.
I've tried a 
>>>>>>>>>>>> couple of
>>>>>>>>>>>> things and none seem to work. Any suggestions
welcome.
>>>>>>>>>>> Mark, did you try failureproperty parameter for
junit task?
>>>>>>>>>>> We may add it to ant sub-targets to raise a flag,
for example,
>>>>>>>> create
>>>>>>>>>> file "
>>>>>>>>>>> TESTS.FAILED" in the root, when tests for some
module fail. 
>>>>>>>>>>> And in
>>>>>>>> the
>>>>>>>>>> end
>>>>>>>>>>> of tests suite run check whether this file exists
on not.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Stepan.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>> Mark.
>>>>>>>>>>>>
>>>>>>>>>>>> On 4/11/06, Stepan Mishura <stepan.mishura@gmail.com>
wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've checked out at morning last updates,
built the code base
>>>>>>>> and
>>>>>>>>>> run
>>>>>>>>>>>> the
>>>>>>>>>>>>> tests …and there are 24 tests failures!
>>>>>>>>>>>>>
>>>>>>>>>>>>> There are 9 tests failures in
>>>>>>>>>>>>> org.apache.harmony.tests.java.nio.channels.DatagramChannelTest

>>>>>>>>>>>>> –
>>>>>>>> I
>>>>>>>>>> saw
>>>>>>>>>>>> these
>>>>>>>>>>>>> failures before from time to time. It
seems that tests 
>>>>>>>>>>>>> depend on
>>>>>>>>>> some
>>>>>>>>>>>> race
>>>>>>>>>>>>> conditions because they may pass if I
rerun them. Paulex, are
>>>>>>>> these
>>>>>>>>>>>> tests
>>>>>>>>>>>>> passing for you?
>>>>>>>>>>>>>
>>>>>>>>>>>>> And there are new 15 test failures. So
now if I'll make a 
>>>>>>>>>>>>> code
>>>>>>>>>> update
>>>>>>>>>>>> or a
>>>>>>>>>>>>> bug fix how I can be 100% sure that I
don't do any 
>>>>>>>>>>>>> regression?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Currently if a commit breaks the Harmony
classlib build a
>>>>>>>>>> notification
>>>>>>>>>>>> with
>>>>>>>>>>>>> subject: "[continuum] BUILD FAILURE:
Classlib/linux.ia32" 
>>>>>>>>>>>>> will
>>>>>>>> be
>>>>>>>>>> send
>>>>>>>>>>>> to
>>>>>>>>>>>>> harmony-commits mailing list. Is it possible
to have the same
>>>>>>>> for
>>>>>>>>>> tests?
>>>>>>>>>>>> I
>>>>>>>>>>>>> mean that after completing automatic
build the existing 
>>>>>>>>>>>>> classlib
>>>>>>>>>> tests
>>>>>>>>>>>> suite
>>>>>>>>>>>>> should be run. If there are failing tests
then a report
>>>>>>>> notification
>>>>>>>>>>>> with
>>>>>>>>>>>>> corresponding subject will be send.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Stepan Mishura
>>>>>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>>>> -- 
>>>>>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>
>>>>>>>>>>>> Terms of use : 
>>>>>>>>>>>> http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>>>> -- 
>>>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>
>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Stepan Mishura
>>>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>>
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>> For additional commands, e-mail: 
>>>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: 
>>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Stepan Mishura
>>>>>>>>> Intel Middleware Products Division
>>>>>>>>>
>>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Mark Hindess <mark.hindess@googlemail.com>
>>>>>>>> IBM Java Technology Centre, UK.
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: 
>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> ---------------------------------------------------------------------

>>>>>>>
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: 
>>>>>>> harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: 
>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Stepan Mishura
>>>>>>> Intel Middleware Products Division
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------

>>>>>>
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: 
>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: 
>>>>> harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message