harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Qiu" <sean.xx....@gmail.com>
Subject Re: [buildtest] pass rate definition
Date Thu, 18 Oct 2007 12:52:55 GMT
I am puzzle by this issues too.
For normal application, the passrate is easy to estimate.
While for harmony, we have a reference implementation to follow up.

There is no doubt that "BuggyTest", "IntermittentlyFailingTest",
"UnsupportedTest" ,"FailingTest" should  be taken into our consideration.

The only question in issue is about "FailingReferenceTest" (does this mean
no-bug-difference here?).I prefer to consider them as the test's bugs rather
than our bugs. And it is accepable for most of the case.

But if there is broad consensus on this issue between developers, whose
details are not defined by reference, we still need modify our
implementation to comply with the consensus to make the test pass.

Anyway, we can take "FailingReferenceTest" as the test's bug rather than our
implementation. Since we comply with the reference anyway, this should be
the most important criteria. So my suggestion is :

Passrate = 1 - ( "BuggyTest"+ "IntermittentlyFailingTest" +
"UnsupportedTest" + "FailingTest" )  / Total

2007/10/18, Alexei Fedotov <alexei.fedotov@gmail.com>:
> Hello,
> I'm involved in numerous discussions on the subject, and want to make
> these discussions transparent to those community members who are
> interested. Imagine we have a test suite which contains five following
> tests:
> BuggyTest.java
>     The test fails due to the test bug.
> FailingReferenceTest.java
>     The test fails on Harmony and passes on RI. The test design does
> not imply that the test should pass.
> IntermittentlyFailingTest.java
>     The test fails intermittently due to HDK bug.
> UnsupportedTest.java
>     The test produces an expected fail due to unimplemented
> functionality in HDK.
> FailingTest.java
>     The test fails due to HDK bug.
> PassingTest.java
>     This one prints PASSED and completes successfully.
> What would be the correct formula to define a pass rate? All agree
> that the rate is a number of passed tests divided to a total number of
> tests. Then people start to argue what are the numerator and the
> denominator.
> One may say, that he counts any failures as bugs. Then she gets 16.66%
> pass rate. Others get 50%, ignoring all fail reasons except the one
> which produces a fixable HDK failure.
> If anyone could share common sense knowledge or Apache practices on
> the subject, this would be interesting.
> --
> With best regards,
> Alexei,
> ESSD, Intel

Sean Qiu
China Software Development Lab, IBM

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message