cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-10017) Add quick-test attribute to cherry picked tests in the smoke tests suite
Date Thu, 11 Oct 2018 09:50:00 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-10017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646211#comment-16646211
] 

ASF GitHub Bot commented on CLOUDSTACK-10017:
---------------------------------------------

borisstoyanov commented on issue #2209: CLOUDSTACK-10017: Adding 'quick-test' attribute to
list of smoketests…
URL: https://github.com/apache/cloudstack/pull/2209#issuecomment-428893141
 
 
   I've just rebased @rhtyd, a full smoketest would take day+ and this label gives you a real
quick feedback if your changes brake anything fundamentally. 
   
   Actually in theory smoketests should be really quick and should give near instant result
back. Their main outcome should be yes/no answer to 'is the build testable' question (as per
common testing practice). I think this is where we got this wrong and kept on adding and adding
more items to the suite, instead it should be the other way around, having a really short
smoketest and more extensive regression testing suite.  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Add quick-test attribute to cherry picked tests in the smoke tests suite
> ------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-10017
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-10017
>             Project: CloudStack
>          Issue Type: Improvement
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: marvin
>            Reporter: Boris Stoyanov
>            Priority: Major
>
> As a first step we can work with the test attributes and introduce a “quick-test”
tag. This tag would be used on tests that are vital and really high level, like for example
CRUD operations and some basic functionality like deploy a VM. The idea behind that is to
execute this for less then an hour and get quick feedback on your changes. Based on this feedback
you’ll be able to take decision whether to continue using this build for testing or not.

> Currently smoketests run for about a day. And surely we can skip some of the areas since
they are not touched. For example if you did a change in the VR, there’s no need to run
all the Account tests on this PR, but focus more on the tests that are relative to the change.
Using the same approach we can divide all the tests into categories, for example: vm-deployment,
virtual-router, accounts, storage, infrastructure, projects, events and so on… Having this
categories we would have more focused test execution for our PRs and it’ll make it more
faster. 
> For example if I submit a PR for a change in the virtual router. As a first step I would
execute the “quick-test” label, this would give me early feedback that there’s generally
nothing wrong with my changes and most of the CloudStack feature are working (<1h in time).
Then I would run the tests with the label “virtual-router” which would drill down the
directory and fetch me the tests related to the VR only and execute them. 
> This approach could save about half the time of test execution or even more… 
> Nevertheless these changes would bring no change in the way people are using the tests
right now and they will be able to execute them as before. It’s more like a on demand asking
for a group of tests to be executed. 
> As next steps we could think of a way to utilize the labels in GitHub more and link them
with the test categories and specific hypervisors. After that we could look for a way to automate
the test runs with BlueOrangutan with the given labels. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message