harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Fedotov" <alexei.fedo...@gmail.com>
Subject [buildtest] pass rate definition
Date Thu, 18 Oct 2007 11:30:29 GMT
Hello,

I'm involved in numerous discussions on the subject, and want to make
these discussions transparent to those community members who are
interested. Imagine we have a test suite which contains five following
tests:

BuggyTest.java
    The test fails due to the test bug.

FailingReferenceTest.java
    The test fails on Harmony and passes on RI. The test design does
not imply that the test should pass.

IntermittentlyFailingTest.java
    The test fails intermittently due to HDK bug.

UnsupportedTest.java
    The test produces an expected fail due to unimplemented
functionality in HDK.

FailingTest.java
    The test fails due to HDK bug.

PassingTest.java
    This one prints PASSED and completes successfully.

What would be the correct formula to define a pass rate? All agree
that the rate is a number of passed tests divided to a total number of
tests. Then people start to argue what are the numerator and the
denominator.

One may say, that he counts any failures as bugs. Then she gets 16.66%
pass rate. Others get 50%, ignoring all fail reasons except the one
which produces a fixable HDK failure.

If anyone could share common sense knowledge or Apache practices on
the subject, this would be interesting.


-- 
With best regards,
Alexei,
ESSD, Intel

Mime
View raw message