ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolay Izhikov <nizhi...@apache.org>
Subject Re: Brainstorm: Make TC Run All faster
Date Mon, 29 Apr 2019 09:03:32 GMT
> Let's imagine that we have an infinite number of agents

Why we should imagine it?
We don't have infinite number of agents.

And we have several concurrent Run All.


В Пн, 29/04/2019 в 11:50 +0300, Павлухин Иван пишет:
> Vyacheslav,
> 
> I finally figured out that "faster" means "total agent time".
> 
> Let's imagine that we have an infinite number of agents. And 2 approaches:
> 1. Uber "Build Apache Ignite" containing all checks.
> 2. Separate jobs for compilation, checkstyle and etc.
> 
> 1st approach will take less agent time in sum. 2nd approach will
> complete faster in wall clock time. And your main concern is "total
> agent time". Am I right?
> 
> пн, 29 апр. 2019 г. в 11:42, Vyacheslav Daradur <daradurvs@gmail.com>:
> > 
> > Hi, Ivan,
> > 
> > We are in the thread "Make TC Run All faster", so the main benefit is
> > to make TC faster :)
> > 
> > Advantages:
> > - 1 TC agent required instead of 4;
> > - RunAll will be faster, in case of builds queue;
> > 
> > Also, the "licenses" profile is included in the step of a release
> > build. I believe check-style also should be included.
> > 
> > The generation of Javadocs is an optional step at preparing the
> > release, but its check on TC takes significant time in case of the
> > separate build.
> > 
> > > > Returning to "Build Apache Ignite" it seems to me that ideally, it can
> > 
> > be hierarchical.
> > 
> > I agree, all the checks may be set as a separate step in the build's
> > configuration. This helps with the main problem I'm trying to solve -
> > resolving of dependencies which takes the most time of the builds.
> > 
> > On Mon, Apr 29, 2019 at 11:24 AM Павлухин Иван <vololo100@gmail.com>
wrote:
> > > 
> > > Vyacheslav, Maxim,
> > > 
> > > Can we once again outline what benefits aggregated "Build Apache
> > > Ignite" performing various checks has comparing to a modularized
> > > approach in which separate builds perform separate tasks?
> > > 
> > > For example, modularized approach looks nice because it is similar to
> > > good practices in software development where we separate
> > > responsibilities between different classes instead of aggregating them
> > > into a single class. And as usual multiple classes works together
> > > coordinating by a class from upper level. So, in fact it is a
> > > hierarchical structure.
> > > 
> > > Returning to "Build Apache Ignite" it seems to me that ideally it can
> > > be hierarchical. There is a top level compilation (assembly?) job but
> > > it is always clear what tasks does it perform (check style, check
> > > license and other subjobs).
> > > 
> > > пт, 26 апр. 2019 г. в 17:06, Maxim Muzafarov <maxmuzaf@gmail.com>:
> > > > 
> > > > Folks,
> > > > 
> > > > +1 for merging all these suites into the single one. All these suites
> > > > (Build Apache Ignite, Javadoc, Licenses Header, Checkstyle) required
> > > > to be `green` all the time. So, we can consider making them a part of
> > > > build Apache Ignite procedure.
> > > > 
> > > > Also, I'd suggest going deeper. We can try to merge `Licenses Header`
> > > > into the `Code style checker` [1]. This will simplify the code
> > > > checking process.
> > > > 
> > > > [1] http://checkstyle.sourceforge.net/config_header.html
> > > > 
> > > > On Fri, 26 Apr 2019 at 13:17, Vyacheslav Daradur <daradurvs@gmail.com>
wrote:
> > > > > 
> > > > > Ivan, you are right, I meant to combine them into one.
> > > > > 
> > > > > Here is a build [1], with enabled profiles (check-licenses,
> > > > > checkstyle) and check of javadoc to show the idea.
> > > > > 
> > > > > Seems it takes ~15 minutes.
> > > > > 
> > > > > [1] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ExperimentalBuildApacheIgniteJavadocLicensesHeaderCheckstyle&branch_IgniteTests24Java8=<default>;
> > > > > 
> > > > > On Fri, Apr 26, 2019 at 12:06 PM Павлухин Иван <vololo100@gmail.com>
wrote:
> > > > > > 
> > > > > > Hi Vyacheslav,
> > > > > > 
> > > > > > What do you mean by uniting?
> > > > > > 
> > > > > > For me it looks like [Javadocs] and [Check Code Style] are not
so time
> > > > > > consuming comparing to tests, are not they? Do you suggest to
combine
> > > > > > mentioned 4 jobs into one? How long will it run in a such case?
> > > > > > 
> > > > > > чт, 25 апр. 2019 г. в 10:50, Vyacheslav Daradur <daradurvs@gmail.com>:
> > > > > > > 
> > > > > > > Hi Igniters,
> > > > > > > 
> > > > > > > At the moment we have several separated test suites:
> > > > > > > * ~Build Apache Ignite~ _ ~10..20mins
> > > > > > > * [Javadocs] _ ~10mins
> > > > > > > * [Licenses Headers] _ ~1min
> > > > > > > * [Check Code Style] _ ~7min
> > > > > > > The most time of each build (except Licenses Headers) is
taken by
> > > > > > > dependency resolving.
> > > > > > > 
> > > > > > > Their main goal is a check that the project is built properly.
> > > > > > > 
> > > > > > > Also, profiles of [Javadocs], [Licenses Headers] uses at
the step of
> > > > > > > preparing release (see DEVNOTES.txt) that means they are
important.
> > > > > > > 
> > > > > > > I'd suggest uniting the builds, this should reduce the
time of tests
> > > > > > > on ~15 minutes and releases agents.
> > > > > > > 
> > > > > > > What do you think?
> > > > > > > 
> > > > > > > On Tue, Nov 27, 2018 at 3:56 PM Павлухин Иван
<vololo100@gmail.com> wrote:
> > > > > > > > 
> > > > > > > > Roman,
> > > > > > > > 
> > > > > > > > Do you have some expectations how faster "correlated"
tests
> > > > > > > > elimination will make Run All? Also do you have a
vision how can we
> > > > > > > > determine such "correlated" tests, can we do it relatively
fast?
> > > > > > > > 
> > > > > > > > But all in all, I am not sure that reducing a group
of correlated
> > > > > > > > tests to only one test can show good stability.
> > > > > > > > пн, 26 нояб. 2018 г. в 17:48, aplatonov <aplatonovv@gmail.com>:
> > > > > > > > > 
> > > > > > > > > It should be noticed that additional parameter
TEST_SCALE_FACTOR was added.
> > > > > > > > > This parameter with ScaleFactorUtil methods can
be used for test size
> > > > > > > > > scaling for different runs (like ordinary and
nightly RunALLs). If someone
> > > > > > > > > want to distinguish these builds he/she can apply
scaling methods from
> > > > > > > > > ScaleFactorUtil in own tests. For nightly test
TEST_SCALE_FACTOR=1.0, for
> > > > > > > > > non-nightly builds TEST_SCALE_FACTOR<1.0.
For example in
> > > > > > > > > GridAbstractCacheInterceptorRebalanceTest test
ScaleFactorUtil was used for
> > > > > > > > > scaling count of iterations. I guess that TEST_SCALE_FACTOR
support will be
> > > > > > > > > added to runs at the same time with RunALL (nightly)
runs.
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > --
> > > > > > > > Best regards,
> > > > > > > > Ivan Pavlukhin
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > --
> > > > > > > Best Regards, Vyacheslav D.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > --
> > > > > > Best regards,
> > > > > > Ivan Pavlukhin
> > > > > 
> > > > > 
> > > > > 
> > > > > --
> > > > > Best Regards, Vyacheslav D.
> > > 
> > > 
> > > 
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > 
> > 
> > 
> > --
> > Best Regards, Vyacheslav D.
> 
> 
> 

Mime
View raw message