activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadrian Zbarcea (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-5552) introduce a smoke-test profile that is enabled by default and during release:prepare
Date Sat, 31 Jan 2015 03:09:34 GMT

    [ https://issues.apache.org/jira/browse/AMQ-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299612#comment-14299612
] 

Hadrian Zbarcea commented on AMQ-5552:
--------------------------------------

I agree with some of the comments, I am mostly against this proposal. Will try to explain
why.

I agree for instance that running tests for hours unreliably gives a bad first impression.
Quite horrible, I'd think, and not only first impression, second and third too.

This proposal only addresses releases from the title of this issue. Time is not a problem
as this is a one time excercise. Any entity/company who uses internal builds have no problem
running it overnight, however long it takes, as long as the result is pretty much guaranteed.
For us as a community is it enough to say that it's a good release it passed the smoke testing?
I would say not, the ASF guidelines say the same thing. If the PMC votes an official release,
we guarantee that it actually works, which is demonstrated by the tests that pass. Those that
don't should be fixed or removed (if they serve no purpose).

I understand the need to be pragmatic. However, even if we run the tests with -fae, if tests
fail in a <module>, tests in all other modules that depend on it will not be run. Therefore
failing tests may hide other failing tests, very hard or impractical to discover. There is
another practical reason. At some point, I introduced a regression while merging to a branch.
One of the best ways to identify the source is to use git bisect. Given the number of commits
I needed some 7 iterations. Now with activemq, where it takes some 10 mins for a build skipping
tests, plus running the test(s) you're interested, it would take at least 15-20 min per iteration.
While a significant chunk of time, at least it's feasible. If the tests you start with are
unreliable, it's a much, much harder effort.

There are other arguments, community related. 

I don't know what would constitute resolution of this issue. In my opinon certainly not something
that would be at odds with the ASF guidelines for releases, or at least they way I understand
them.

> introduce a smoke-test profile that is enabled by default and during release:prepare
> ------------------------------------------------------------------------------------
>
>                 Key: AMQ-5552
>                 URL: https://issues.apache.org/jira/browse/AMQ-5552
>             Project: ActiveMQ
>          Issue Type: Improvement
>          Components: Distribution, Test Cases
>    Affects Versions: 5.11.0
>            Reporter: Gary Tully
>              Labels: distribution, mvn, smoke, tests, validation
>             Fix For: 5.12.0
>
>
> Users should be able to do $>mvn install on trunk or the source distribution and get
a validated (smoke-tested) distribution in < 10  minutes.
> The smoke-test profile should be enabled for release:prepare
> At the moment, more than 3k tests are run, they are not reliable enough and the build
is gone for a number of hours. This gives a bad first impression.
> Or course we should continue to improve the test suite but this has a totally different
focus.
> The smoke-test profile takes a smart cross section of tests in each module that validate
core functionality. 
> It will be an interesting challenge to get the selection right; balancing typical use
cases with coverage with speed etc.
> The tests should be:
>  * representative of the module functionality
>  * clean - no hard-coded ports and use only space on target
>  * fast
>  * reliable
>  * can be run in parallel (maybe if it allows more tests to be run in the same time frame)
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message