cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From runseb <...@git.apache.org>
Subject [GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...
Date Thu, 22 Oct 2015 10:50:36 GMT
Github user runseb commented on the pull request:

    https://github.com/apache/cloudstack/pull/960#issuecomment-150178317
  
    I think you guys agree on the problem but disagree on the solution.
    To me 100% success on BVT, if it includes ~20% (or whatever percentage) of skipped tests
is not a real 100% success rate on BVT.
    
    So @miguelaferreira prefers to see 80% success on BVT and vote +1 on a release, if he
actually knows what's happening with the 20% failure.
    While @nitt10prashant and @pvr9711 you seem to want 100% success on BVT even if it includes
some tests that are actually not run.
    
    The nicira test is actually the only one in there that's testing a third party plugin.
I don't see any tests for solidfire, netscaler, open contrail, midokura etc….All theses
tests will fail without the proper hardware/software setup and hence will fail with simulator
run.
    
    If we really want 100% on integration tests (so called BVT) we need to define a test suite
that everyone can run without special hardware or software. And those tests *cannot* be skipped.
    
    I personally fail to see the benefit of skipped tests versus a failed test.
    
    For this particular tests I propose we move it to a folder /test/integration/plugins and
have our base tests defining "BVT" as the test in /test/integration/smoke
    
    So -1 on this PR
    +1 on a PR that would move this test to a separate folder.
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message