ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roman Kondakov <kondako...@mail.ru.INVALID>
Subject Re: Brainstorm: Make TC Run All faster
Date Sun, 25 Nov 2018 18:26:47 GMT
Hi Dmitriy!

We have over 50 000 test in our tests base. And this number will be 
noticeably increased soon by MVCC tests coverage activity. This means 
that it is very difficult to rework and rewrite these test manually to 
make it run faster. But we can choose another way. Do we have an ability 
to perform a statistical analysis over a considerable number of last 
tests runs? If we do, let's consider two points:

1. After careful consideration in terms of statistics it may turnout 
that significant number of these test are "evergreen" tests. It means 
that these tests check cases which are very difficult to break. If so, 
why should we run these tests each time? They are great candidates for 
night runs.

2. After dropping "evergreen" tests there are may be a number of tests 
with correlated results. There could be a lot of test groups with a some 
number of tests in each group, where either all tests are red or all 
tests are green. In this case in "fast" runs we can launch only one test 
from each group instead of entire group. Other tests in group can be 
launched at night build.


Having a list of "good" tests (good tests = all tests - evergreen tests 
- groups (except chosen represenative from each group)), we can mark 
these tests with annotation @Category (or @Tag in junit 5). For fast 
tests runs we can run only annotated tests, for night runs - all tests 
as usual.

When new test is added developers could decide to add or not to add this 
annotation.

Annotated tests list should be reviewed monthly or weekly. Or, if 
possible, automate this procedure.


-- 
Kind Regards
Roman Kondakov

On 15.11.2018 13:34, Dmitriy Pavlov wrote:
> Hi Igniters,
>
>
>
> Some of us started to use the Bot to get an approval of PRs. It helps to
> protect master from new failures, but this requires to run RunAll tests set
> for each commit and this makes markable pressure to TC infra.
>
>
>
> I would like to ask you to share your ideas on how to make runAll faster,
> maybe you can share any of your measurements and any other info about
> (possible) bottlenecks.
>
>
>
> Sincerely,
>
> Dmitriy Pavlov
>

Mime
View raw message