activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From artnaseef <...@artnaseef.com>
Subject [DISCUSS] Improving ActiveMQ Tests
Date Sun, 01 Feb 2015 01:25:04 GMT
*Background*
As we all know, ActiveMQ tests take a very long time to execute, and a
number of the tests have been unreliable.  Personally, it was ActiveMQ that
lead me to learn maven's "skip test" features early on because I would not
wait hours to get a small change into a build during development, nor did I
want to spend time (as a user of ActiveMQ - this is before I was a
committer) trying to resolve test failures.  This contradicts industry
best-practices.
A large, world-class product such as ActiveMQ relies, wisely, on automated
tests to ensure regression bugs are not introduced with new feature
additions and bug fixes.
ActiveMQ unit tests really contains a variety of test types, such as:
 Use Case tests
 Load tests
 Bug Reproduction tests
 Unit Tests that target a single class (i.e. True Unit Tests)*
* I personally haven't seen any of these in ActiveMQ, but I suspect/hope
that some exist.

*Goals*
Improve the automated testing with the following result in mind:
Developer cycle time: reduce the amount of time to execute tests when making
changes to ActiveMQ code so that a developer gets a reasonable expectation
that changes are valid without drawing the development cycle out into
excessive time periods
 Continued quality: continue to cover all of the test scenarios currently
covered
 Release validation: continue to provide very thorough validation for
releases at the expense of significant, additional test time
 Reliability: improve reliability of tests for the release process so that
it is reasonable to expect a good build every time for quality releases
 Verify units: improve overall quality of the system through more thorough
testing of each unit (class) so that small bugs in units do not turn into
hard-to-find bugs in the system (I found one of these just walking through
code yesterday - it will get fixed soon)

*Ideas*
 Identify junit tests that are really integration or load tests and separate
as such  
   
 Create an artifact for integration tests, move integration tests there and
run them during the verify phase  
 Create an artifact for load tests, move load tests there and also run
during the verify phase (interim)  
 Stand up a true load-test environment and move load tests into that
environment and out of the main build process
 Create true unit tests with high code coverage
 Move tests into separate artifacts to reduce the number of tests, and
therefore the amount of time necessary to resume tests from a failure point
(e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc)
 Review problematic tests and determine ways to resolve (e.g. replace with
new tests using different approaches, use the load test environment, etc)

*Wrap-Up*
The steps above will move ActiveMQ testing toward a solid, reliable test
framework without creating gaps in quality.
Unless alternative plans are defined, I am going to enter Jira tickets for
these and start moving in this direction.  Volunteers are welcome and
greatly appreciated.



--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message