activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Frazier <mmfraz...@me.com>
Subject Re: [DISCUSS] Improving ActiveMQ Tests
Date Sun, 01 Feb 2015 01:35:20 GMT
+1, this seems like a great idea Art.

I'll be glad to help with this.


> On Jan 31, 2015, at 5:25 PM, artnaseef <art@artnaseef.com> wrote:
> 
> *Background*
> As we all know, ActiveMQ tests take a very long time to execute, and a
> number of the tests have been unreliable.  Personally, it was ActiveMQ that
> lead me to learn maven's "skip test" features early on because I would not
> wait hours to get a small change into a build during development, nor did I
> want to spend time (as a user of ActiveMQ - this is before I was a
> committer) trying to resolve test failures.  This contradicts industry
> best-practices.
> A large, world-class product such as ActiveMQ relies, wisely, on automated
> tests to ensure regression bugs are not introduced with new feature
> additions and bug fixes.
> ActiveMQ unit tests really contains a variety of test types, such as:
> Use Case tests
> Load tests
> Bug Reproduction tests
> Unit Tests that target a single class (i.e. True Unit Tests)*
> * I personally haven't seen any of these in ActiveMQ, but I suspect/hope
> that some exist.
> 
> *Goals*
> Improve the automated testing with the following result in mind:
> Developer cycle time: reduce the amount of time to execute tests when making
> changes to ActiveMQ code so that a developer gets a reasonable expectation
> that changes are valid without drawing the development cycle out into
> excessive time periods
> Continued quality: continue to cover all of the test scenarios currently
> covered
> Release validation: continue to provide very thorough validation for
> releases at the expense of significant, additional test time
> Reliability: improve reliability of tests for the release process so that
> it is reasonable to expect a good build every time for quality releases
> Verify units: improve overall quality of the system through more thorough
> testing of each unit (class) so that small bugs in units do not turn into
> hard-to-find bugs in the system (I found one of these just walking through
> code yesterday - it will get fixed soon)
> 
> *Ideas*
> Identify junit tests that are really integration or load tests and separate
> as such  
> 
> Create an artifact for integration tests, move integration tests there and
> run them during the verify phase  
> Create an artifact for load tests, move load tests there and also run
> during the verify phase (interim)  
> Stand up a true load-test environment and move load tests into that
> environment and out of the main build process
> Create true unit tests with high code coverage
> Move tests into separate artifacts to reduce the number of tests, and
> therefore the amount of time necessary to resume tests from a failure point
> (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc)
> Review problematic tests and determine ways to resolve (e.g. replace with
> new tests using different approaches, use the load test environment, etc)
> 
> *Wrap-Up*
> The steps above will move ActiveMQ testing toward a solid, reliable test
> framework without creating gaps in quality.
> Unless alternative plans are defined, I am going to enter Jira tickets for
> these and start moving in this direction.  Volunteers are welcome and
> greatly appreciated.
> 
> 
> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Mime
View raw message