cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remi Bergsma <RBerg...@schubergphilis.com>
Subject Re: [DISCUSS] Move to Github
Date Sun, 20 Dec 2015 09:17:11 GMT
Hi Rene,

We’re at the stage that running the integration tests is a one-liner and 7 hour wait [1].
It’s running a subset [2] of available Marvin tests.

The problem is that Marvin is slow and sometimes flaky (error found, rerun is fine). So the
test results need manual review before you can publish them. This alone is a lot of work.

The biggest issue is that there is no community hardware that can run this and there is no
way to hook tests into Github either (unless we get full access).

About LGTM, I usually want one based on the integration tests and one based on code review.

Regards,
Remi

[1] https://github.com/schubergphilis/MCT-shared/blob/master/helper_scripts/cloudstack/check-pr.sh



[2] https://github.com/schubergphilis/MCT-shared/blob/master/helper_scripts/cloudstack/run_marvin_router_tests.sh


On 19/12/15 23:11, "Rene Moser" <mail@renemoser.net> wrote:

>
>
>On 12/19/2015 07:57 PM, Remi Bergsma wrote:
>
>> I disagree with testing based on complexity. You simply cannot know the implications
upfront, as that is why you run the tests. What seems small, can break it all.
>> 
>> Example:
>> This commit seems an easy_fix, right? Just a findbugs issue resolved.
>> https://github.com/apache/cloudstack/commit/6a4927f660f776bcbd12ae45f4e63ae2c2e96774
>
>Ok, definition of easy is for _me_: fixing a typo in a debug log output,
>adding license headers, fixing a comment. adding documentation.
>
>So we just have minor and major, fine for me. But would you run
>integration tests for adding missing license header?
>
>> It was just merged indeed, exactly as you propose. But it did cause a major outage.
And I don’t even use HyperV.
>> Details: https://github.com/apache/cloudstack/pull/761
>> 
>> This is why all PRs to 4.6 and 4.7 (200+) that we merged over the last couple of
months were tested against a real cloud. No easy fixes and minor changes. We always need to
run the full integration tests.
>
>This would be the best case (and good job btw!) but for how long can you
>made it without automation? We need automated tests, and if possible
>full blown automated integration testing. I agree.
>
>> When new functionality is proposed, there aren’t many people willing to write unit
and integration tests to cover it. Until that changes, testing can only guard whatever the
tests cover. And when we merge new stuff without tests, the total coverage goes down making
the tests less relevant. In fact, when we resolve a bug we should write a tests along with
it. I know of one guy that does that on a regular basis.
>
>It is ~2016. No excuse. I know we can not test everything, we are
>dealing with hardware. But I would rather say, "merge it, it covers
>tests and they passed" then "merge it, it has 2 LGTM".
>
>> It’s not so simple as it seems unfortunately.
>
>It has never been simple, and I didn't say so, but it should be our goal
>to get there. Right?
>
>Regards
>René
Mime
View raw message