harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Varlamov" <alexey.v.varla...@gmail.com>
Subject Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT
Date Thu, 28 Dec 2006 07:56:02 GMT
2006/12/28, Alexander Kleymenov <kleymenov@gmail.com>:
> > BTW, Mikhail, are we going to integrate IR-level JIT regression tests
> > ([1]) into the *DRLVM regression testing infrastructure* in the
> > nearest future?
>
> Stop, guys, let's think. Regression test is a test checking for
> particular problem reported and fixed before. So regression test and
> JIRA report have one-to-one correspondence. What I see at HARMONY-1531
> is a set of tests for Jitrino.OPT marked as "improvement", not as a bug.
> So, I think, these tests should go to different place, not to regression
> test suite. Right?

Alexander,
I'm not expert in testing taxonomy, could you please share the full
picture - which tests go where and when/how intended to be run, etc?

> > Are there any pending tasks to reach that point? IMHO, we should
> > integrate/commit it and let go for constant improvement. That's quite
> > enough for the start. (Not the highest priority, but gonnabe pretty
> > useful). Please, tell me if I am missing something here.
>
> I'm agree with you. Tests first! Any tests increase the quality
> of the product (if they do not do some intentional harm of course).

I got an impression that great diversity of test suites may come into
conflict with this manifested goal - more frameworks to maintain, more
steps to launch/track various tests, scattered reports, more complex
desicion tree to add new tests, etc.
Leaving aside official certification tests, I personally do not really
care where from a test case is taken - from spec perseption of code
implementer or from real-life user problem, the essence point is that
it properly exercises particular bit of functionality. And further
grading IMO should be practical tradeoff between
importance/coverage/time to test, dividing pre-commit/pre-contribute
and nightly or weekly CC runs.
Ideally there should be just one entry point which takes in a
config/testbase to be tested and a single consolidated test status and
report.

--
Alexey

Mime
View raw message