db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Myrna van Lunteren (JIRA)" <j...@apache.org>
Subject [jira] Commented: (DERBY-1116) Define a minimal acceptance test suite for checkins
Date Fri, 04 Feb 2011 00:38:23 GMT

    [ https://issues.apache.org/jira/browse/DERBY-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990412#comment-12990412
] 

Myrna van Lunteren commented on DERBY-1116:
-------------------------------------------

I propose to close this issue. Of course, this was essentially something that applied to the
old derbyall when it held all tests...with lots of waiting around in between. But it could
be extended to the junit suites.All, which takes more than Rick's desired 1 hour. 

However, even if we apply this to suites.All, I think we shouldn't do this. 
Firstly, I think creating a minimal subset will cause extra complexity and maintenance trouble.
We'll have to go through and figure out exceptionally good tests, and communally decide on
it and adjust the tests in it...Then maintain that list/suite to ensure it's still accurate.
Perhaps that's something we can look at again once we got Emma working again, but even then
it would be lower on *my* list than adding tests in areas we don't properly cover now.
Secondly, which tests to run is different for each change. In principle, we should run the
entire suite when in doubt - that's why we have the regression tests - but for many changes
that is just obviously overkill. And on the other hand, no matter which tests we put in a
minimal acceptance test, there will be changes that would necessitate other tests to be run...
Finally, with the recent (since 10.5) changes enabling running of junit suites or runs in
parallel I think the overall time for the tests can be configured to an acceptable time.


> Define a minimal acceptance test suite for checkins
> ---------------------------------------------------
>
>                 Key: DERBY-1116
>                 URL: https://issues.apache.org/jira/browse/DERBY-1116
>             Project: Derby
>          Issue Type: Improvement
>          Components: Test
>            Reporter: David Van Couvering
>            Priority: Minor
>
> Now that we have an excellent notification system for tinderbox/nightly regression failures,
I would like to suggest that we reduce the size of the test suite being run prior to checkin.
  I am not sure what should be in such a minimal test, but in particular I would like to remove
things such as the stress test and generally reduce the number of tests being run for each
subsystem/area of code.
> As an example of how derbyall currently affects my productivity, I was running derbyall
on my machine starting at 2pm, and by evening it was still running.  At 9pm my machine was
accidentally powered down, and this morning I am restarting the test run.
> I have been tempted (and acted on such temptation) in the past to run a smaller set of
tests, only to find out that I have blocked others who are running derbyall prior to checkin.
 For this reason, we need to define a minimal acceptance test (MATS) that we all agree to
run prior to checkin.
> One could argue that you can run your tests on another machine and thus reduce productivity,
but we can't assume everybody in the community has nice big test servers to run their tests
on.
> If there are no objections, I can take a first pass at defining what this test suite
should look like, but I suspect many others in the community have strong opinions about this
and may even wish to volunteer to do this definition themselves (for example, some of you
who may be working in the QA division in some of our Big Companies :) ).

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message