river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Costers <jonathan.cost...@googlemail.com>
Subject Re: ServiceDiscoveryManager test coverage
Date Thu, 26 Aug 2010 23:22:54 GMT
There is one now already: qa.run
We just need to pass it more test category names (property run.categories)
by default.
Maybe I am misunderstanding you?

2010/8/27 Peter Firmstone <jini@zeus.net.au>

> Hi JC,
>
> Can we have an ant target for running all the tests?
>
> And how about a qa.run.hudson target?
>
> I usually use run-categories, to isolate what I'm working on, but we
> definitely need a target that runs everything that should be, even if it
> does take overnight.
>
> Regards,
>
> Peter.
>
>
> Jonathan Costers wrote:
>
>> 2010/8/24 Patricia Shanahan <pats@acm.org>
>>
>>
>>
>>> On 8/22/2010 4:57 PM, Peter Firmstone wrote:
>>> ...
>>>
>>>  Thanks Patricia, that's very helpful, I'll figure it out where I went
>>>
>>>
>>>> wrong this week, it really shows the importance of full test coverage.
>>>>
>>>>
>>>>
>>> ...
>>>
>>> I strongly agree that test coverage is important. Accordingly, I've done
>>> some analysis of the "ant qa.run" output.
>>>
>>> There are 1059 test description (*.td) files that exist, and are loaded
>>> at
>>> the start of "ant qa.run", but that do not seem to be run. I've extracted
>>> the top level categories from those files:
>>>
>>> constraint
>>> discoveryproviders_impl
>>> discoveryservice
>>> end2end
>>> eventmailbox
>>> export_spec
>>> io
>>> javaspace
>>> jeri
>>> joinmanager
>>> jrmp
>>> loader
>>> locatordiscovery
>>> lookupdiscovery
>>> lookupservice
>>> proxytrust
>>> reliability
>>> renewalmanager
>>> renewalservice
>>> scalability
>>> security
>>> start
>>> txnmanager
>>>
>>> I'm sure some of these tests are obsolete, duplicates of tests in
>>> categories that are being run, or otherwise inappropriate, but there does
>>> seem to be a rich vein of tests we could mine.
>>>
>>>
>>>
>>
>> The QA harness loads all .td files under the "spec" and "impl" directories
>> when starting and only witholds the ones that are tagged with the
>> categories
>> that we specify from the Ant target.
>> Whenever a test is really obsolete or otherwise not supposed to run, it is
>> marked with a "SkipTestVerifier" in its .td file.
>> Most of these are genuine and should be run though.
>> There are more categories than the ones you mention above, for instance:
>> "spec", "id", "id_spec", etc.
>> Also, some tests are tagged with multiple categories and as such
>> duplicates
>> can exist when assembling the list of tests to run.
>>
>> The reason not all of them are run (by Hudson) now is that we give a
>> specific set of test categories that are known (to me) to run smoothly.
>> There are many others that are not run (by default) because issue(s) are
>> present with one or more of the tests in that category.
>>
>> I completely agree with the fact that we should not exclude complete test
>> categories because of one test failing.
>> What we probably should do is tag any problematic test (due to
>> infrastructure or other reasons) with a SkipTestVerifier for the time
>> being
>> so that it is not taken into account by the QA harness for now.
>> That way, we can add all test categories to the default Ant run.
>> However, this would take a large amount of time to run (I've tried it
>> once,
>> and killed the process after several days), which brings us to your next
>> point:
>>
>> Part of the problem may be time to run the tests. I'd like to propose
>>
>>
>>> splitting the tests into two sets:
>>>
>>> 1. A small set that one would run in addition to the relevant tests,
>>> whenever making a small change. It should *not* be based on skipping
>>> complete categories, but on doing those tests from each category that are
>>> most likely to detect regression, especially regression due to changes in
>>> other areas.
>>>
>>>
>>>
>>
>> Completely agree. However, most of the QA tests are not clear unit or
>> regression tests. They are more integration/conformance tests that test
>> the
>> requirements of the spec and its implementation.
>> Identifying the list of "right" tests to run as part of the small set you
>> mention would require going through all 1059 test descriptions and their
>> sources.
>>
>> 2. A full test set that may take a lot longer. In many projects, there is
>> a
>>
>>
>>> "nightly build" and a test sequence that is run against that build. That
>>> test sequence can take up to 24 hours to run, and should be as complete
>>> as
>>> possible. Does Apache have infrastructure to support this sort of
>>> operation?
>>>
>>>
>>>
>>
>> Again, completely agree. I'm sure Apache supports this through Hudson. We
>> could request to setup a second build job, doing nightly builds and
>> running
>> the whole test suite. Think this is the only way to make running the
>> complete QA suite automatically practical.
>>
>>
>>
>>
>>> Are there any tests that people *know* should not run? I'm thinking of
>>> running the lot just to see what happens, but knowing ones that are not
>>> expected to work would help with result interpretation.
>>>
>>>
>>>
>>
>> See above, tests of that type should have already been tagged to be
>> skipped
>> by the good people that donated this test suite.
>> I've noticed that usually, when a SkipTestVerifier is used in a .td file,
>> someone has put some comments in there to explain why it was tagged as
>> such.
>>
>>
>>
>>
>>> Patricia
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message