activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jamie G." <jamie.goody...@gmail.com>
Subject Re: [DISCUSS] Improving ActiveMQ Tests
Date Sun, 01 Feb 2015 21:13:22 GMT
+1 Art,

I had some other ideas on another thread, reading the general concept
it sounds like a possible way forward to making AMQ testing easier.

Cheers,
J

On Sun, Feb 1, 2015 at 12:34 AM, Clebert Suconic
<clebert.suconic@gmail.com> wrote:
> We have been through that exercise already on activemq-6. Perhaps it
> would be a better use of resources if we worked towards activemq-6?
>
> With hornetq (now activemq-6) we differentiated unit-tests,
> integration tests, soak tests and performance tests.
>
> When you do a build you won't do the entire testsuite to validate a
> simple build.. it's just a matter of minutes and simple unit tests.
>
> With integration tests (we could find a better name for it) we do more
> extensive tests and even then we limit certain things as good
> practices. (like using more latches and less Sleeps.. unless you
> really have to. It's impossible to run all the profiles on every
> time.. Like you can't force someone to run a 10Hours soak test to make
> a simple build... we have profiles for that.
>
> Best practices is the key on keeping the testsuite clean. As I said on
> my first paragraph, maybe we should concentrate efforts on activemq-6,
> Improving what we have on activemq-6 repo is a better bet IMO.
>
> We can keep this discussion open if you need help on any specifics on
> coding and running the profiles we have.
>
>
> On Sat, Jan 31, 2015 at 8:35 PM, Mark Frazier <mmfrazier@me.com> wrote:
>> +1, this seems like a great idea Art.
>>
>> I'll be glad to help with this.
>>
>>
>>> On Jan 31, 2015, at 5:25 PM, artnaseef <art@artnaseef.com> wrote:
>>>
>>> *Background*
>>> As we all know, ActiveMQ tests take a very long time to execute, and a
>>> number of the tests have been unreliable.  Personally, it was ActiveMQ that
>>> lead me to learn maven's "skip test" features early on because I would not
>>> wait hours to get a small change into a build during development, nor did I
>>> want to spend time (as a user of ActiveMQ - this is before I was a
>>> committer) trying to resolve test failures.  This contradicts industry
>>> best-practices.
>>> A large, world-class product such as ActiveMQ relies, wisely, on automated
>>> tests to ensure regression bugs are not introduced with new feature
>>> additions and bug fixes.
>>> ActiveMQ unit tests really contains a variety of test types, such as:
>>> Use Case tests
>>> Load tests
>>> Bug Reproduction tests
>>> Unit Tests that target a single class (i.e. True Unit Tests)*
>>> * I personally haven't seen any of these in ActiveMQ, but I suspect/hope
>>> that some exist.
>>>
>>> *Goals*
>>> Improve the automated testing with the following result in mind:
>>> Developer cycle time: reduce the amount of time to execute tests when making
>>> changes to ActiveMQ code so that a developer gets a reasonable expectation
>>> that changes are valid without drawing the development cycle out into
>>> excessive time periods
>>> Continued quality: continue to cover all of the test scenarios currently
>>> covered
>>> Release validation: continue to provide very thorough validation for
>>> releases at the expense of significant, additional test time
>>> Reliability: improve reliability of tests for the release process so that
>>> it is reasonable to expect a good build every time for quality releases
>>> Verify units: improve overall quality of the system through more thorough
>>> testing of each unit (class) so that small bugs in units do not turn into
>>> hard-to-find bugs in the system (I found one of these just walking through
>>> code yesterday - it will get fixed soon)
>>>
>>> *Ideas*
>>> Identify junit tests that are really integration or load tests and separate
>>> as such
>>>
>>> Create an artifact for integration tests, move integration tests there and
>>> run them during the verify phase
>>> Create an artifact for load tests, move load tests there and also run
>>> during the verify phase (interim)
>>> Stand up a true load-test environment and move load tests into that
>>> environment and out of the main build process
>>> Create true unit tests with high code coverage
>>> Move tests into separate artifacts to reduce the number of tests, and
>>> therefore the amount of time necessary to resume tests from a failure point
>>> (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc)
>>> Review problematic tests and determine ways to resolve (e.g. replace with
>>> new tests using different approaches, use the load test environment, etc)
>>>
>>> *Wrap-Up*
>>> The steps above will move ActiveMQ testing toward a solid, reliable test
>>> framework without creating gaps in quality.
>>> Unless alternative plans are defined, I am going to enter Jira tickets for
>>> these and start moving in this direction.  Volunteers are welcome and
>>> greatly appreciated.
>>>
>>>
>>>
>>> --
>>> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Improving-ActiveMQ-Tests-tp4690766.html
>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic
> http://community.jboss.org/people/clebert.suconic@jboss.com
> http://clebertsuconic.blogspot.com

Mime
View raw message