incubator-ooo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Weir <robw...@apache.org>
Subject Re: [RELEASE] Planning QA activities (was: Dictionary extensions)
Date Sun, 15 Apr 2012 22:40:13 GMT
On Sun, Apr 15, 2012 at 6:22 PM, Andrea Pescetti <pescetti@apache.org> wrote:
> On 15/04/2012 Rob Weir wrote:
>>
>> On Sun, Apr 15, 2012 at 11:49 AM, Andrea Pescetti wrote:
>>>
>>> Historically, OpenOffice.org produced a numbered Release Candidate
>>> (OpenOffice.org 3.3 had ten, RC1 to RC10) that was made available to the
>>> community exactly for the purpose of looking for unknown bugs. QA
>>> activities
>>> were planned for the RC phase and, after a few days of availability, a RC
>>> was approved as final or rejected (and the Release Manager could, and at
>>> times did, include fixes for previously known bugs that had been
>>> accumulating and that were significant; none of them was a blocker in
>>> itself, but each of them would have caused problems to users, so their
>>> combined effect was blocking the release).
>>
>> That is what we've been doing with the dev snapsots builds.  Surely
>> you've seen the QA work Lily has done with testing them?
>
>
> Sure, and it was great work. But those tests were run on versions that are
> now quite outdated. Example: the spell check test asks you to verify that no
> spell check is available and that dictionaries have been removed accurately,
> while we all know that things are quite different in current builds.
>
> The traditional OpenOffice.org Quality Assurance process had two features
> that I'd like to keep:
>
> 1) Full QA is run on what we release. We need to ensure that OpenOffice
> works now, not that previous builds worked.
>

"Full" to me means a test matrix on all platforms, all functions, all
languages, and includes performance, interop and other tests.  A full
test pass probably takes 4-6 weeks if we do it right.  I don't think
we have the opportunity to do more than one "full" test pass per
release.   But this should happen much earlier, right after a "feature
freeze" when all features targeted for a release have been checked in.

> 2) The community at large is involved in testing, in a specific short period
> just before the release. If "Release Candidate" has a different meaning now,
> call it the "Pre-Release Phase", that will end with a "Release Candidate" we
> can vote on rather confidently. We can't rely on volunteers doing QA on a
> regular basis, but we can very easily find volunteers willing to stress-test
> a "nearly final" version in a well defined timeframe (1-2 weeks).
>
> I would keep this as a basis for discussing how to structure QA activities
> in future (after 3.4), if others agree.
>

This is only a fraction of the picture.  We also need to consider what
changes are allowed, and how those changes are reviewed, as we
progress toward a Release Candidate.  That is a critical part of how
we maintain quality.  If committers are checking in whatever they feel
like, up to the day we have a RC, then we will have serious quality
problems.

When we changing code we "invalidate" prior testing.  Not in an
absolute sense, for most code changes there is an test impact, and
some portion of the "full test" needs to be redone, to verify both the
fix, but also that we did not break anything else.  Again, this
requires discipline from committers.  "Last minute" feature work, or
even undisciplined bug fixing, can cause more problems than it fixes.

So I think we want the trunk to be fee CTR up to a certain point
(feature freeze?) and after that point only changes that are
accompanied by a BZ issue are allowed, but still CTR.  Then at some
point, after the main test pass is complete, we start prioritizing
bugs as we create snapshot builds.  We gradually raise the level
required for bugs to be accepted.  P3, P2, P1.  In the final weeks we
fix only marked release blockers.   Once we have the RC build we
branch the code.  At that point (and maybe even earlier) we switch to
RTC, e.g., no changes are allowed unless they are reviewed first.

Was anything like this done before with OOo?

-Rob

> Regards,
>  Andrea.

Mime
View raw message