ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: Interesting statistics for our tests
Date Sat, 17 Dec 2016 08:36:50 GMT
Dima,
I do not think we really need more hardware for now. 40 min test will not
run faster if we add more agents. We have a problem with tests quality, not
quantity.

Rohit,
Yes, Ignite is ver advanced framework, so we have about 80-90 different
suites, testing different parts of API.

Vladimir.

On Sat, Dec 17, 2016 at 2:47 AM, Rohit Mohta <rmohta.coder@gmail.com> wrote:

> Interesting numbers, to say the least :)
>
> Separating test suites, as suggested by Vladimir sounds good.
>
> Probably off-placed question:
>    As Ignite provides multiple features and is suitable for multiple use
> case- should we categorize test suites based on features?
>
> ------
> Rohit Mohta
>
> > On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <dsetrakyan@apache.org>
> wrote:
> >
> > In addition to optimizing the test suites, should we add more TC agents
> as
> > well?
> >
> > On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <vozerov@gridgain.com>
> > wrote:
> >
> >> And one more thing - how much time is spent on duplicated "Binary"
> suites?
> >> *15
> >> hours*, or almost *40%* of total "Run All" execution time. It means,
> that
> >> if move suites for *OptimizedMarshaller *(which we consider deprecated)
> >> into a separate circuit (which will be executed not per-branch, but e.g.
> >> nightly) we will almost double peak TeamCity throughput.
> >>
> >> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <vozerov@gridgain.com>
> >> wrote:
> >>
> >>> Igniters,
> >>>
> >>> I collected several thought-provoking numbers for our TeamCity
> >>> (config-variations tests are excluded):
> >>>
> >>> * Number of Java suites: 78
> >>> * Total execution time without build overhead: 38.5 hours
> >>> * Total test classes: 3376
> >>> * Total tests: 60681
> >>>
> >>> * 25% of time is consumed by 881 tests (41 test class)
> >>> * 50% of time is consumed by 2755 tests (153 test class)
> >>> * 75% of time is consumed by 15911 tests (443 test class)
> >>>
> >>> Or to be more demonstrative:
> >>> * 1.5% of tests consume 25% of time
> >>> * 4.5% of tests consume 50% of time
> >>> * 26.2% tests consume 75% of time
> >>>
> >>> TOP 3 test classes:
> >>> * CacheSerializableTransactionsTest - 46 minutes
> >>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> >>> * GridCachePartitionedNodeRestartTest - 25 minutes
> >>>
> >>> Morale:
> >>> 1) We have lots of poor failover/concurrency tests requiring
> >>> disproportional time to run;
> >>> 2) If we split all tests into "core" and "failover/concurrency" parts
> >>> properly, then *core* part will take *~15 mins* to answer you with high
> >>> level of confidence whether your change broken anything or not.
> >>>
> >>> Thoughts?
> >>>
> >>> Vladimir.
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message