ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yakov Zhdanov <yzhda...@apache.org>
Subject Re: Interesting statistics for our tests
Date Sat, 17 Dec 2016 08:47:14 GMT
Agree with Vladimir's suggestions. Great analysis!

--Yakov

2016-12-17 11:36 GMT+03:00 Vladimir Ozerov <vozerov@gridgain.com>:

> Dima,
> I do not think we really need more hardware for now. 40 min test will not
> run faster if we add more agents. We have a problem with tests quality, not
> quantity.
>
> Rohit,
> Yes, Ignite is ver advanced framework, so we have about 80-90 different
> suites, testing different parts of API.
>
> Vladimir.
>
> On Sat, Dec 17, 2016 at 2:47 AM, Rohit Mohta <rmohta.coder@gmail.com>
> wrote:
>
> > Interesting numbers, to say the least :)
> >
> > Separating test suites, as suggested by Vladimir sounds good.
> >
> > Probably off-placed question:
> >    As Ignite provides multiple features and is suitable for multiple use
> > case- should we categorize test suites based on features?
> >
> > ------
> > Rohit Mohta
> >
> > > On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <dsetrakyan@apache.org>
> > wrote:
> > >
> > > In addition to optimizing the test suites, should we add more TC agents
> > as
> > > well?
> > >
> > > On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <vozerov@gridgain.com
> >
> > > wrote:
> > >
> > >> And one more thing - how much time is spent on duplicated "Binary"
> > suites?
> > >> *15
> > >> hours*, or almost *40%* of total "Run All" execution time. It means,
> > that
> > >> if move suites for *OptimizedMarshaller *(which we consider
> deprecated)
> > >> into a separate circuit (which will be executed not per-branch, but
> e.g.
> > >> nightly) we will almost double peak TeamCity throughput.
> > >>
> > >> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <
> vozerov@gridgain.com>
> > >> wrote:
> > >>
> > >>> Igniters,
> > >>>
> > >>> I collected several thought-provoking numbers for our TeamCity
> > >>> (config-variations tests are excluded):
> > >>>
> > >>> * Number of Java suites: 78
> > >>> * Total execution time without build overhead: 38.5 hours
> > >>> * Total test classes: 3376
> > >>> * Total tests: 60681
> > >>>
> > >>> * 25% of time is consumed by 881 tests (41 test class)
> > >>> * 50% of time is consumed by 2755 tests (153 test class)
> > >>> * 75% of time is consumed by 15911 tests (443 test class)
> > >>>
> > >>> Or to be more demonstrative:
> > >>> * 1.5% of tests consume 25% of time
> > >>> * 4.5% of tests consume 50% of time
> > >>> * 26.2% tests consume 75% of time
> > >>>
> > >>> TOP 3 test classes:
> > >>> * CacheSerializableTransactionsTest - 46 minutes
> > >>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> > >>> * GridCachePartitionedNodeRestartTest - 25 minutes
> > >>>
> > >>> Morale:
> > >>> 1) We have lots of poor failover/concurrency tests requiring
> > >>> disproportional time to run;
> > >>> 2) If we split all tests into "core" and "failover/concurrency" parts
> > >>> properly, then *core* part will take *~15 mins* to answer you with
> high
> > >>> level of confidence whether your change broken anything or not.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> Vladimir.
> > >>>
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message