activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clebert Suconic <clebert.suco...@gmail.com>
Subject Re: [DISCUSS] Improving ActiveMQ Tests
Date Sun, 01 Feb 2015 04:04:59 GMT
We have been through that exercise already on activemq-6. Perhaps it
would be a better use of resources if we worked towards activemq-6?

With hornetq (now activemq-6) we differentiated unit-tests,
integration tests, soak tests and performance tests.

When you do a build you won't do the entire testsuite to validate a
simple build.. it's just a matter of minutes and simple unit tests.

With integration tests (we could find a better name for it) we do more
extensive tests and even then we limit certain things as good
practices. (like using more latches and less Sleeps.. unless you
really have to. It's impossible to run all the profiles on every
time.. Like you can't force someone to run a 10Hours soak test to make
a simple build... we have profiles for that.

Best practices is the key on keeping the testsuite clean. As I said on
my first paragraph, maybe we should concentrate efforts on activemq-6,
Improving what we have on activemq-6 repo is a better bet IMO.

We can keep this discussion open if you need help on any specifics on
coding and running the profiles we have.


On Sat, Jan 31, 2015 at 8:35 PM, Mark Frazier <mmfrazier@me.com> wrote:
> +1, this seems like a great idea Art.
>
> I'll be glad to help with this.
>
>
>> On Jan 31, 2015, at 5:25 PM, artnaseef <art@artnaseef.com> wrote:
>>
>> *Background*
>> As we all know, ActiveMQ tests take a very long time to execute, and a
>> number of the tests have been unreliable.  Personally, it was ActiveMQ that
>> lead me to learn maven's "skip test" features early on because I would not
>> wait hours to get a small change into a build during development, nor did I
>> want to spend time (as a user of ActiveMQ - this is before I was a
>> committer) trying to resolve test failures.  This contradicts industry
>> best-practices.
>> A large, world-class product such as ActiveMQ relies, wisely, on automated
>> tests to ensure regression bugs are not introduced with new feature
>> additions and bug fixes.
>> ActiveMQ unit tests really contains a variety of test types, such as:
>> Use Case tests
>> Load tests
>> Bug Reproduction tests
>> Unit Tests that target a single class (i.e. True Unit Tests)*
>> * I personally haven't seen any of these in ActiveMQ, but I suspect/hope
>> that some exist.
>>
>> *Goals*
>> Improve the automated testing with the following result in mind:
>> Developer cycle time: reduce the amount of time to execute tests when making
>> changes to ActiveMQ code so that a developer gets a reasonable expectation
>> that changes are valid without drawing the development cycle out into
>> excessive time periods
>> Continued quality: continue to cover all of the test scenarios currently
>> covered
>> Release validation: continue to provide very thorough validation for
>> releases at the expense of significant, additional test time
>> Reliability: improve reliability of tests for the release process so that
>> it is reasonable to expect a good build every time for quality releases
>> Verify units: improve overall quality of the system through more thorough
>> testing of each unit (class) so that small bugs in units do not turn into
>> hard-to-find bugs in the system (I found one of these just walking through
>> code yesterday - it will get fixed soon)
>>
>> *Ideas*
>> Identify junit tests that are really integration or load tests and separate
>> as such
>>
>> Create an artifact for integration tests, move integration tests there and
>> run them during the verify phase
>> Create an artifact for load tests, move load tests there and also run
>> during the verify phase (interim)
>> Stand up a true load-test environment and move load tests into that
>> environment and out of the main build process
>> Create true unit tests with high code coverage
>> Move tests into separate artifacts to reduce the number of tests, and
>> therefore the amount of time necessary to resume tests from a failure point
>> (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc)
>> Review problematic tests and determine ways to resolve (e.g. replace with
>> new tests using different approaches, use the load test environment, etc)
>>
>> *Wrap-Up*
>> The steps above will move ActiveMQ testing toward a solid, reliable test
>> framework without creating gaps in quality.
>> Unless alternative plans are defined, I am going to enter Jira tickets for
>> these and start moving in this direction.  Volunteers are welcome and
>> greatly appreciated.
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



-- 
Clebert Suconic
http://community.jboss.org/people/clebert.suconic@jboss.com
http://clebertsuconic.blogspot.com

Mime
View raw message