river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Firmstone <j...@zeus.net.au>
Subject Re: ServiceDiscoveryManager test coverage
Date Thu, 26 Aug 2010 23:33:34 GMT
Ok, I'm happy if we do that then, I just want to make sure I'm running 
all the tests I should, that's all, so qa.run should run all categories 
by default?

Peter.

Jonathan Costers wrote:
> There is one now already: qa.run
> We just need to pass it more test category names (property run.categories)
> by default.
> Maybe I am misunderstanding you?
>
> 2010/8/27 Peter Firmstone <jini@zeus.net.au>
>
>   
>> Hi JC,
>>
>> Can we have an ant target for running all the tests?
>>
>> And how about a qa.run.hudson target?
>>
>> I usually use run-categories, to isolate what I'm working on, but we
>> definitely need a target that runs everything that should be, even if it
>> does take overnight.
>>
>> Regards,
>>
>> Peter.
>>
>>
>> Jonathan Costers wrote:
>>
>>     
>>> 2010/8/24 Patricia Shanahan <pats@acm.org>
>>>
>>>
>>>
>>>       
>>>> On 8/22/2010 4:57 PM, Peter Firmstone wrote:
>>>> ...
>>>>
>>>>  Thanks Patricia, that's very helpful, I'll figure it out where I went
>>>>
>>>>
>>>>         
>>>>> wrong this week, it really shows the importance of full test coverage.
>>>>>
>>>>>
>>>>>
>>>>>           
>>>> ...
>>>>
>>>> I strongly agree that test coverage is important. Accordingly, I've done
>>>> some analysis of the "ant qa.run" output.
>>>>
>>>> There are 1059 test description (*.td) files that exist, and are loaded
>>>> at
>>>> the start of "ant qa.run", but that do not seem to be run. I've extracted
>>>> the top level categories from those files:
>>>>
>>>> constraint
>>>> discoveryproviders_impl
>>>> discoveryservice
>>>> end2end
>>>> eventmailbox
>>>> export_spec
>>>> io
>>>> javaspace
>>>> jeri
>>>> joinmanager
>>>> jrmp
>>>> loader
>>>> locatordiscovery
>>>> lookupdiscovery
>>>> lookupservice
>>>> proxytrust
>>>> reliability
>>>> renewalmanager
>>>> renewalservice
>>>> scalability
>>>> security
>>>> start
>>>> txnmanager
>>>>
>>>> I'm sure some of these tests are obsolete, duplicates of tests in
>>>> categories that are being run, or otherwise inappropriate, but there does
>>>> seem to be a rich vein of tests we could mine.
>>>>
>>>>
>>>>
>>>>         
>>> The QA harness loads all .td files under the "spec" and "impl" directories
>>> when starting and only witholds the ones that are tagged with the
>>> categories
>>> that we specify from the Ant target.
>>> Whenever a test is really obsolete or otherwise not supposed to run, it is
>>> marked with a "SkipTestVerifier" in its .td file.
>>> Most of these are genuine and should be run though.
>>> There are more categories than the ones you mention above, for instance:
>>> "spec", "id", "id_spec", etc.
>>> Also, some tests are tagged with multiple categories and as such
>>> duplicates
>>> can exist when assembling the list of tests to run.
>>>
>>> The reason not all of them are run (by Hudson) now is that we give a
>>> specific set of test categories that are known (to me) to run smoothly.
>>> There are many others that are not run (by default) because issue(s) are
>>> present with one or more of the tests in that category.
>>>
>>> I completely agree with the fact that we should not exclude complete test
>>> categories because of one test failing.
>>> What we probably should do is tag any problematic test (due to
>>> infrastructure or other reasons) with a SkipTestVerifier for the time
>>> being
>>> so that it is not taken into account by the QA harness for now.
>>> That way, we can add all test categories to the default Ant run.
>>> However, this would take a large amount of time to run (I've tried it
>>> once,
>>> and killed the process after several days), which brings us to your next
>>> point:
>>>
>>> Part of the problem may be time to run the tests. I'd like to propose
>>>
>>>
>>>       
>>>> splitting the tests into two sets:
>>>>
>>>> 1. A small set that one would run in addition to the relevant tests,
>>>> whenever making a small change. It should *not* be based on skipping
>>>> complete categories, but on doing those tests from each category that are
>>>> most likely to detect regression, especially regression due to changes in
>>>> other areas.
>>>>
>>>>
>>>>
>>>>         
>>> Completely agree. However, most of the QA tests are not clear unit or
>>> regression tests. They are more integration/conformance tests that test
>>> the
>>> requirements of the spec and its implementation.
>>> Identifying the list of "right" tests to run as part of the small set you
>>> mention would require going through all 1059 test descriptions and their
>>> sources.
>>>
>>> 2. A full test set that may take a lot longer. In many projects, there is
>>> a
>>>
>>>
>>>       
>>>> "nightly build" and a test sequence that is run against that build. That
>>>> test sequence can take up to 24 hours to run, and should be as complete
>>>> as
>>>> possible. Does Apache have infrastructure to support this sort of
>>>> operation?
>>>>
>>>>
>>>>
>>>>         
>>> Again, completely agree. I'm sure Apache supports this through Hudson. We
>>> could request to setup a second build job, doing nightly builds and
>>> running
>>> the whole test suite. Think this is the only way to make running the
>>> complete QA suite automatically practical.
>>>
>>>
>>>
>>>
>>>       
>>>> Are there any tests that people *know* should not run? I'm thinking of
>>>> running the lot just to see what happens, but knowing ones that are not
>>>> expected to work would help with result interpretation.
>>>>
>>>>
>>>>
>>>>         
>>> See above, tests of that type should have already been tagged to be
>>> skipped
>>> by the good people that donated this test suite.
>>> I've noticed that usually, when a SkipTestVerifier is used in a .td file,
>>> someone has put some comments in there to explain why it was tagged as
>>> such.
>>>
>>>
>>>
>>>
>>>       
>>>> Patricia
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>         
>>>
>>>       
>>     
>
>   


Mime
View raw message