river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Firmstone <j...@zeus.net.au>
Subject Re: ServiceDiscoveryManager test coverage
Date Fri, 27 Aug 2010 22:20:54 GMT
Hmm that is strange, I thought that was svn based activity, I don't 
recall accessing svn then either.

Is someone able to look into it?

Jonathan Costers wrote:
> That's strange ... Hudson lists a user "Peter Firmstone" with last activity
> 4 days ago.
> The user that was setup for me last year unfortunately disappeared from the
> list :-(
>
> 2010/8/27 Peter Firmstone <jini@zeus.net.au>
>
>   
>> Ok, sounds good, I don't have access to Hudson unfortunately though, can
>> anyone assist here?
>>
>>
>> Jonathan Costers wrote:
>>
>>     
>>> Ideally it should, yes. So we get visibility on failures sooner ...
>>>
>>> To accomplish that, we need to make a change to the property value of
>>> run.categories and add the missing categories (I've already added two more
>>> the last cpl days).
>>> However, adding all other categories to the run.categories property will
>>> make our next Hudson builds take a very, very long time.
>>>
>>> So, as proposed above, we could (request to) setup a separate Hudson job
>>> that does this complete test run nightly, instead of triggered by SVN
>>> changes.
>>> - the current Hudson job would build triggered by SVN changes and run some
>>> unit tests and/or some selected QA tests (fast)
>>> - a new Hudson job would do nighly (or otherwise scheduled) builds and run
>>> the whole QA suite (very slow)
>>>
>>> How does that sound to you?
>>>
>>> Speaking of Hudson, any chance we can get that up and running again? I see
>>> nothing has been built in a while.
>>>
>>> 2010/8/27 Peter Firmstone <jini@zeus.net.au>
>>>
>>>
>>>
>>>       
>>>> Ok, I'm happy if we do that then, I just want to make sure I'm running
>>>> all
>>>> the tests I should, that's all, so qa.run should run all categories by
>>>> default?
>>>>
>>>> Peter.
>>>>
>>>>
>>>> Jonathan Costers wrote:
>>>>
>>>>
>>>>
>>>>         
>>>>> There is one now already: qa.run
>>>>> We just need to pass it more test category names (property
>>>>> run.categories)
>>>>> by default.
>>>>> Maybe I am misunderstanding you?
>>>>>
>>>>> 2010/8/27 Peter Firmstone <jini@zeus.net.au>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> Hi JC,
>>>>>>
>>>>>> Can we have an ant target for running all the tests?
>>>>>>
>>>>>> And how about a qa.run.hudson target?
>>>>>>
>>>>>> I usually use run-categories, to isolate what I'm working on, but
we
>>>>>> definitely need a target that runs everything that should be, even
if
>>>>>> it
>>>>>> does take overnight.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Peter.
>>>>>>
>>>>>>
>>>>>> Jonathan Costers wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> 2010/8/24 Patricia Shanahan <pats@acm.org>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> On 8/22/2010 4:57 PM, Peter Firmstone wrote:
>>>>>>>> ...
>>>>>>>>
>>>>>>>>  Thanks Patricia, that's very helpful, I'll figure it out
where I
>>>>>>>> went
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> wrong this week, it really shows the importance of full
test
>>>>>>>>> coverage.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> ...
>>>>>>>>
>>>>>>>> I strongly agree that test coverage is important. Accordingly,
I've
>>>>>>>> done
>>>>>>>> some analysis of the "ant qa.run" output.
>>>>>>>>
>>>>>>>> There are 1059 test description (*.td) files that exist,
and are
>>>>>>>> loaded
>>>>>>>> at
>>>>>>>> the start of "ant qa.run", but that do not seem to be run.
I've
>>>>>>>> extracted
>>>>>>>> the top level categories from those files:
>>>>>>>>
>>>>>>>> constraint
>>>>>>>> discoveryproviders_impl
>>>>>>>> discoveryservice
>>>>>>>> end2end
>>>>>>>> eventmailbox
>>>>>>>> export_spec
>>>>>>>> io
>>>>>>>> javaspace
>>>>>>>> jeri
>>>>>>>> joinmanager
>>>>>>>> jrmp
>>>>>>>> loader
>>>>>>>> locatordiscovery
>>>>>>>> lookupdiscovery
>>>>>>>> lookupservice
>>>>>>>> proxytrust
>>>>>>>> reliability
>>>>>>>> renewalmanager
>>>>>>>> renewalservice
>>>>>>>> scalability
>>>>>>>> security
>>>>>>>> start
>>>>>>>> txnmanager
>>>>>>>>
>>>>>>>> I'm sure some of these tests are obsolete, duplicates of
tests in
>>>>>>>> categories that are being run, or otherwise inappropriate,
but there
>>>>>>>> does
>>>>>>>> seem to be a rich vein of tests we could mine.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> The QA harness loads all .td files under the "spec" and "impl"
>>>>>>> directories
>>>>>>> when starting and only witholds the ones that are tagged with
the
>>>>>>> categories
>>>>>>> that we specify from the Ant target.
>>>>>>> Whenever a test is really obsolete or otherwise not supposed
to run,
>>>>>>> it
>>>>>>> is
>>>>>>> marked with a "SkipTestVerifier" in its .td file.
>>>>>>> Most of these are genuine and should be run though.
>>>>>>> There are more categories than the ones you mention above, for
>>>>>>> instance:
>>>>>>> "spec", "id", "id_spec", etc.
>>>>>>> Also, some tests are tagged with multiple categories and as such
>>>>>>> duplicates
>>>>>>> can exist when assembling the list of tests to run.
>>>>>>>
>>>>>>> The reason not all of them are run (by Hudson) now is that we
give a
>>>>>>> specific set of test categories that are known (to me) to run
>>>>>>> smoothly.
>>>>>>> There are many others that are not run (by default) because issue(s)
>>>>>>> are
>>>>>>> present with one or more of the tests in that category.
>>>>>>>
>>>>>>> I completely agree with the fact that we should not exclude complete
>>>>>>> test
>>>>>>> categories because of one test failing.
>>>>>>> What we probably should do is tag any problematic test (due to
>>>>>>> infrastructure or other reasons) with a SkipTestVerifier for
the time
>>>>>>> being
>>>>>>> so that it is not taken into account by the QA harness for now.
>>>>>>> That way, we can add all test categories to the default Ant run.
>>>>>>> However, this would take a large amount of time to run (I've
tried it
>>>>>>> once,
>>>>>>> and killed the process after several days), which brings us to
your
>>>>>>> next
>>>>>>> point:
>>>>>>>
>>>>>>> Part of the problem may be time to run the tests. I'd like to
propose
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> splitting the tests into two sets:
>>>>>>>>
>>>>>>>> 1. A small set that one would run in addition to the relevant
tests,
>>>>>>>> whenever making a small change. It should *not* be based
on skipping
>>>>>>>> complete categories, but on doing those tests from each category
that
>>>>>>>> are
>>>>>>>> most likely to detect regression, especially regression due
to
>>>>>>>> changes
>>>>>>>> in
>>>>>>>> other areas.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> Completely agree. However, most of the QA tests are not clear
unit or
>>>>>>> regression tests. They are more integration/conformance tests
that
>>>>>>> test
>>>>>>> the
>>>>>>> requirements of the spec and its implementation.
>>>>>>> Identifying the list of "right" tests to run as part of the small
set
>>>>>>> you
>>>>>>> mention would require going through all 1059 test descriptions
and
>>>>>>> their
>>>>>>> sources.
>>>>>>>
>>>>>>> 2. A full test set that may take a lot longer. In many projects,
there
>>>>>>> is
>>>>>>> a
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> "nightly build" and a test sequence that is run against that
build.
>>>>>>>> That
>>>>>>>> test sequence can take up to 24 hours to run, and should
be as
>>>>>>>> complete
>>>>>>>> as
>>>>>>>> possible. Does Apache have infrastructure to support this
sort of
>>>>>>>> operation?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> Again, completely agree. I'm sure Apache supports this through
Hudson.
>>>>>>> We
>>>>>>> could request to setup a second build job, doing nightly builds
and
>>>>>>> running
>>>>>>> the whole test suite. Think this is the only way to make running
the
>>>>>>> complete QA suite automatically practical.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Are there any tests that people *know* should not run? I'm
thinking
>>>>>>>> of
>>>>>>>> running the lot just to see what happens, but knowing ones
that are
>>>>>>>> not
>>>>>>>> expected to work would help with result interpretation.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> See above, tests of that type should have already been tagged
to be
>>>>>>> skipped
>>>>>>> by the good people that donated this test suite.
>>>>>>> I've noticed that usually, when a SkipTestVerifier is used in
a .td
>>>>>>> file,
>>>>>>> someone has put some comments in there to explain why it was
tagged as
>>>>>>> such.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Patricia
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>
>>>>>>>               
>>>>>>             
>>>>>
>>>>>           
>>>>         
>>>
>>>       
>>     
>
>   


Mime
View raw message