ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Павлухин Иван <vololo...@gmail.com>
Subject Re: Brainstorm: Make TC Run All faster
Date Fri, 23 Nov 2018 09:18:05 GMT
Hi,

I have some thoughts about tests speed. First of all I must say that I
do not see any simple and lightweight solution.

I did some measurements a while ago and it looks like that simply
optimizing number of Ignite node start/stop calls will not give us
great speed up. I naively measured [1] time spent in node start/stop
methods and for Cache 1 suite it turns out that only 2.5 minutes is
spent there when whole suite run is about 1 hour. But I made an
experiment only for Cache 1 suite and might have been not accurate
enough.

Moreover I thought about refactoring whole test base to employ some
faster patterns and it looks unfeasible to me to accomplish with it in
relatively short amount of time.

So, I can imagine a following approach. There are a lot of
load/concurrent tests, which executions are limited either by big
number of iterations (1000+) or by time (30 sec, 60 sec). In ideal
world I think there should not be such tests in Run All. If such tests
regularly catch bugs unnoticed by regular tests then a there are areas
which are not covered by regular (functional?) tests and missing tests
could be added. If such assumption holds then load/concurrent tests
should not less likely to fail if functional tests pass and therefore
could be extracted out of Run All. The real world is not ideal so we
can use a mentioned idea of significantly limiting load/concurrent
tests execution time to 5-10 seconds in Run All.

Of course there is a need to make an analysis to find all
load/concurrent tests and measure theirs execution time. I think we
could use some ML/Data analysis tools for that, could not we?

Some additional figures:
1. Number of tests in Run All ~ 50K.
2. Number of test methods in core module ~ 7500.

[1] https://github.com/apache/ignite/pull/5419/files
пн, 19 нояб. 2018 г. в 14:34, Павлухин Иван <vololo100@gmail.com>:
>
> Hi,
>
> I would like to understand following. We are going to make TC green.
> We are going to make TC fast. Are we going to do it in parallel?
> пн, 19 нояб. 2018 г. в 08:39, Petr Ivanov <mr.weider@gmail.com>:
> >
> > Concerning current TeamCity compute capacity — I think we should invest into it’s
stability at first: there are lots of problems associated with
> >  — tests hangs (which now can render agent useless up to 3 days)
> >  — checkout issues (speedup proxy is installed but sometimes even it is not enough
for checkout on 100 agents simultaneously, server side checkout research is required)
> >  — tests architecture (current one relies heavily on large file movement over
network which also can be a bottleneck in case of 100 agents simultaneous download start)
> > And so on.
> >
> > After it additional donation can be discussed.
> >
> > > On 16 Nov 2018, at 17:05, Vyacheslav Daradur <daradurvs@gmail.com> wrote:
> > >
> > > At one of the meetups, Vladimir Ozerov has said that we may specify
> > > and move less risky to be broken tests in separate build plan which
> > > will be executed daily or weekly
> > >
> > > For example tests of compatibility with different JDK versions or
> > > compatibility between Ignite's releases.
> > >
> > > Also, I agree with Denis we should find and remove tests with the same checks.
> > >
> > > By the way, if someone of the community donates TC agents, can this
> > > help to reduce the time?
> > >
> > > On Thu, Nov 15, 2018 at 5:38 PM Yakov Zhdanov <yzhdanov@apache.org> wrote:
> > >>
> > >> Denis, you can go even further. E.g. you can start topology once for the
> > >> full set of single threaded full api cache tests. Each test should start
> > >> cache dynamically and run it logic.
> > >>
> > >> As for me, I would think of splitting RunAll to 2 steps - one containing
> > >> basic tests and another with more complex tests. 2nd step should not start
> > >> (except manually) if 1st step results in any build failure.
> > >>
> > >> --Yakov
> > >
> > >
> > >
> > > --
> > > Best Regards, Vyacheslav D.
> >
>
>
> --
> Best regards,
> Ivan Pavlukhin



-- 
Best regards,
Ivan Pavlukhin

Mime
View raw message