Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of mmfrazier@me.com designates
 17.158.232.236 as permitted sender)
From: Mark Frazier <mmfrazier@me.com>
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: quoted-printable
MIME-version: 1.0 (1.0)
Subject: Re: [DISCUSS] Improving ActiveMQ Tests
Message-id: <94A91EE5-F5B1-4532-A4D6-432FD74D2596@me.com>
Date: Sat, 31 Jan 2015 17:35:20 -0800
References: <1422753904896-4690766.post@n4.nabble.com>
In-reply-to: <1422753904896-4690766.post@n4.nabble.com>
To: "dev@activemq.apache.org" <dev@activemq.apache.org>

+1, this seems like a great idea Art.

I'll be glad to help with this.


> On Jan 31, 2015, at 5:25 PM, artnaseef <art@artnaseef.com> wrote:
>=20
> *Background*
> As we all know, ActiveMQ tests take a very long time to execute, and a
> number of the tests have been unreliable.  Personally, it was ActiveMQ tha=
t
> lead me to learn maven's "skip test" features early on because I would not=

> wait hours to get a small change into a build during development, nor did I=

> want to spend time (as a user of ActiveMQ - this is before I was a
> committer) trying to resolve test failures.  This contradicts industry
> best-practices.
> A large, world-class product such as ActiveMQ relies, wisely, on automated=

> tests to ensure regression bugs are not introduced with new feature
> additions and bug fixes.
> ActiveMQ unit tests really contains a variety of test types, such as:
> Use Case tests
> Load tests
> Bug Reproduction tests
> Unit Tests that target a single class (i.e. True Unit Tests)*
> * I personally haven't seen any of these in ActiveMQ, but I suspect/hope
> that some exist.
>=20
> *Goals*
> Improve the automated testing with the following result in mind:
> Developer cycle time: reduce the amount of time to execute tests when maki=
ng
> changes to ActiveMQ code so that a developer gets a reasonable expectation=

> that changes are valid without drawing the development cycle out into
> excessive time periods
> Continued quality: continue to cover all of the test scenarios currently
> covered
> Release validation: continue to provide very thorough validation for
> releases at the expense of significant, additional test time
> Reliability: improve reliability of tests for the release process so that
> it is reasonable to expect a good build every time for quality releases
> Verify units: improve overall quality of the system through more thorough
> testing of each unit (class) so that small bugs in units do not turn into
> hard-to-find bugs in the system (I found one of these just walking through=

> code yesterday - it will get fixed soon)
>=20
> *Ideas*
> Identify junit tests that are really integration or load tests and separat=
e
> as such =20
>=20
> Create an artifact for integration tests, move integration tests there and=

> run them during the verify phase =20
> Create an artifact for load tests, move load tests there and also run
> during the verify phase (interim) =20
> Stand up a true load-test environment and move load tests into that
> environment and out of the main build process
> Create true unit tests with high code coverage
> Move tests into separate artifacts to reduce the number of tests, and
> therefore the amount of time necessary to resume tests from a failure poin=
t
> (e.g. split activemq-unit-test into activemq-unit-test01, ...02, etc)
> Review problematic tests and determine ways to resolve (e.g. replace with
> new tests using different approaches, use the load test environment, etc)
>=20
> *Wrap-Up*
> The steps above will move ActiveMQ testing toward a solid, reliable test
> framework without creating gaps in quality.
> Unless alternative plans are defined, I am going to enter Jira tickets for=

> these and start moving in this direction.  Volunteers are welcome and
> greatly appreciated.
>=20
>=20
>=20
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUS=
S-Improving-ActiveMQ-Tests-tp4690766.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.