river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Firmstone <j...@zeus.net.au>
Subject Re: QA Test suite refactoring.
Date Mon, 24 Dec 2012 08:47:15 GMT
First stage of refactoring complete.

     [java] -----------------------------------------
     [java]
     [java] # of tests started   = 1412
     [java] # of tests completed = 1412
     [java] # of tests skipped   = 46
     [java] # of tests passed    = 1412
     [java] # of tests failed    = 0
     [java]
     [java] -----------------------------------------
     [java]
     [java]    Date finished:
     [java]       Mon Dec 24 16:54:31 EST 2012
     [java]    Time elapsed:
     [java]       91393 seconds
     [java]

collect-result:

BUILD SUCCESSFUL
Total time: 1,539 minutes 24 seconds

Peter Firmstone wrote:
> I propose a minor change in semantics for TestFactory.  
> TestDescription includes the following:
>
> /**
> * The <code>TestDescription</code> is an object which describes a test;
> * it is a factory for the test object and provides support facilities
> * for setting up and executing the test.
> * <p>
> * The description contains of a set of properties. Some of these
> * properties have well defined names which are used to construct
> * the execution environment of the test. These well defined names are:
> * <ul>
>
> So we could rename TestFactory and setup method to this:
>
> public interface TestExecutionEnvironment {
>
>    /**
>     * Performs construction of execution environment for this test. 
>     * If an exception is thrown, the <code>tearDown</code> method will
>     * still be called.
>     *
>     * @param config the test properties
>     * @throws Exception if construct fails for any reason
>     */
>    Test construct(QAConfig config) throws Exception;
>
>    /**
>     * Tears down any construct that was needed to run the test.  This 
> method is
>     * called even if construct throws an exception, and so must be 
> designed to
>     * be tolerant of test state that is not completely initialized.
>     */
>    void tearDown();
>   }
>
> Peter Firmstone wrote:
>> I've completed the first phase of refactoring, separating out the 
>> environment setup and test run into separate interfaces as per 
>> appended.   This should allow the setup and teardown and execution of 
>> the tests to remain separate concerns.  At present all tests 
>> implement both interfaces, so nothing has really changed, yet.
>>
>> When the qa test run is complete and has passed all tests, I'll 
>> upload the changes for review.
>>
>> The next stage will be the separation of setup and test execution for 
>> the tests that have experienced failures recently in trunk.  This 
>> will simplify test execution and should make it easier to manage 
>> shared state.
>>
>> From there, perhaps we should merge the changes made in trunk back in 
>> incrementally.
>>
>> Since a considerable amount of work is required to refactor all 
>> tests, I suggest we migrate the tests to the new format slowly over 
>> time on an as needed basis.
>>
>> The names of interfaces and methods can be changed if necessary:
>>
>> public interface TestFactory {
>>
>>    /**
>>     * Performs setup for this test.  If an exception is thrown, the
>>     * <code>tearDown</code> method will still be called.
>>     *
>>     * @param config the test properties
>>     * @throws Exception if setup fails for any reason
>>     */
>>    Test setup(QAConfig config) throws Exception;
>>
>>    /**
>>     * Tears down any setup that was needed to run the test.  This 
>> method is
>>     * called even if setup throws an exception, and so must be 
>> designed to
>>     * be tolerant of test state that is not completely initialized.
>>     */
>>    void tearDown();
>>   }
>>
>>
>> public interface Test {
>>    /**
>>     * Failure type. Indicates that a test failed because of a recognized
>>     * environmental problem. These will typically be due to
>>     * intermittent or recurring configuration dependent failures
>>     * which have been determined to result from problems external
>>     * to the product.
>>     */
>>    int ENV = 1;
>>    /**
>>     * Failure type. Indicates that a test failed but the reason for
>>     * failure may be external to the product being tested.
>>     */
>>    int INDEF = 2;
>>    /**
>>     * Failure type indicating no failure occured. This is a special
>>     * value which should be returned by a FailureAnalyzer to indicate
>>     * that the exception represents a passing condition for the test
>>     */
>>    int PASSED = 6;
>>    /**
>>     * Failure type indicating that the test should be rerun.
>>     */
>>    int RERUN = 4;
>>    /**
>>     * Failure type indicating that the test should not be run.
>>     */
>>    int SKIP = 3;
>>    /**
>>     * Failure type. Indicates that a test failed because the product 
>> failed,
>>     * or an unrecognized environmental failure occured.
>>     */
>>    int TEST = 0;
>>    /**
>>     * Failure type (for analyzers) indicating failure not recognized
>>     */
>>    int UNKNOWN = 5;
>>
>>    /**
>>     * Execute the body of the test.
>>     *
>>     * @throws Exception if the test fails for any reason
>>     */
>>    void run() throws Exception;
>>   }
>>
>> Regards,
>>
>> Peter.
>>
>> Peter Firmstone wrote:
>>> Presently most tests have a lot of shared mutable state.
>>>
>>> This isn't helped by the setup(QAConfig) mutator method in the Test 
>>> interface appended.
>>>
>>> An alternative might be:
>>>
>>> public Test setup(QAConfig config) throws Exception;
>>>
>>> This would allow the test to return another Test object, fully 
>>> constructed with immutable state.
>>>
>>> It would also allow the test to create an object that can run 
>>> several other Test objects, rather than relying on inheritance.
>>>
>>> Thoughts?
>>>
>>>
>>>
>>> /**
>>> * This interface must implemented by all tests supported by the test
>>> * harness.  The following sequence of events is performed for each
>>> * test:
>>> * <p><ul>
>>> *   <li> the test class is instantiated and it's no-arg constructor 
>>> called.
>>> *        The constructor will typically perform minimal initialization,
>>> *        since the test does not have access to the test environment
>>> *   <li> the setup method is called, passing the config object.
>>> *        This provides an opportunity for performing any test setup
>>> *        that relies on accessing configuration parameters.
>>> *   <li> the run method is called to run the test
>>> *   <li> the teardown method is called to clean up state or services
>>> *        created by the test. This method is called
>>> *        even if <code>setup</code> or <code>run</code>
throws an 
>>> exception.
>>> * </ul>
>>> */
>>> public interface Test {
>>>
>>>    /**
>>>     * Failure type. Indicates that a test failed because the product 
>>> failed,
>>>     * or an unrecognized environmental failure occured.
>>>     */
>>>    public final static int TEST = 0;
>>>       /**
>>>     * Failure type. Indicates that a test failed because of a 
>>> recognized
>>>     * environmental problem. These will typically be due to
>>>     * intermittent or recurring configuration dependent failures
>>>     * which have been determined to result from problems external
>>>     * to the product.
>>>     */
>>>    public final static int ENV = 1;
>>>       /**
>>>     * Failure type. Indicates that a test failed but the reason for
>>>     * failure may be external to the product being tested.
>>>     */
>>>    public final static int INDEF = 2;
>>>
>>>    /** Failure type indicating that the test should not be run. */
>>>    public final static int SKIP = 3;
>>>
>>>    /** Failure type indicating that the test should be rerun. */
>>>    public final static int RERUN = 4;
>>>
>>>    /** Failure type (for analyzers) indicating failure not 
>>> recognized */
>>>    public final static int UNKNOWN = 5;
>>>
>>>    /**
>>>     * Failure type indicating no failure occured. This is a special
>>>     * value which should be returned by a FailureAnalyzer to indicate
>>>     * that the exception represents a passing condition for the test
>>>     */
>>>    public final static int PASSED = 6;
>>>
>>>    /**
>>>     * Performs setup for this test.  If an exception is thrown, the
>>>     * <code>tearDown</code> method will still be called.
>>>     *
>>>     * @param config the test properties
>>>     * @throws Exception if setup fails for any reason
>>>     */
>>>    public void setup(QAConfig config) throws Exception;
>>>
>>>    /**
>>>     * Execute the body of the test.
>>>     *
>>>     * @throws Exception if the test fails for any reason
>>>     */
>>>    public void run() throws Exception;
>>>
>>>    /**
>>>     * Tears down any setup that was needed to run the test.  This 
>>> method is
>>>     * called even if setup throws an exception, and so must be 
>>> designed to
>>>     * be tolerant of test state that is not completely initialized.
>>>     */
>>>    public void tearDown();
>>> }
>>>
>>>
>>
>>
>
>


Mime
View raw message