incubator-ooo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From J├╝rgen Schmidt <jogischm...@googlemail.com>
Subject Re: [RELEASE] Planning QA activities
Date Mon, 16 Apr 2012 05:06:52 GMT
On 4/16/12 12:40 AM, Rob Weir wrote:
> On Sun, Apr 15, 2012 at 6:22 PM, Andrea Pescetti<pescetti@apache.org>  wrote:
>> On 15/04/2012 Rob Weir wrote:
>>>
>>> On Sun, Apr 15, 2012 at 11:49 AM, Andrea Pescetti wrote:
>>>>
>>>> Historically, OpenOffice.org produced a numbered Release Candidate
>>>> (OpenOffice.org 3.3 had ten, RC1 to RC10) that was made available to the
>>>> community exactly for the purpose of looking for unknown bugs. QA
>>>> activities
>>>> were planned for the RC phase and, after a few days of availability, a RC
>>>> was approved as final or rejected (and the Release Manager could, and at
>>>> times did, include fixes for previously known bugs that had been
>>>> accumulating and that were significant; none of them was a blocker in
>>>> itself, but each of them would have caused problems to users, so their
>>>> combined effect was blocking the release).
>>>
>>> That is what we've been doing with the dev snapsots builds.  Surely
>>> you've seen the QA work Lily has done with testing them?
>>
>>
>> Sure, and it was great work. But those tests were run on versions that are
>> now quite outdated. Example: the spell check test asks you to verify that no
>> spell check is available and that dictionaries have been removed accurately,
>> while we all know that things are quite different in current builds.
>>
>> The traditional OpenOffice.org Quality Assurance process had two features
>> that I'd like to keep:
>>
>> 1) Full QA is run on what we release. We need to ensure that OpenOffice
>> works now, not that previous builds worked.
>>
>
> "Full" to me means a test matrix on all platforms, all functions, all
> languages, and includes performance, interop and other tests.  A full
> test pass probably takes 4-6 weeks if we do it right.  I don't think
> we have the opportunity to do more than one "full" test pass per
> release.   But this should happen much earlier, right after a "feature
> freeze" when all features targeted for a release have been checked in.
>
>> 2) The community at large is involved in testing, in a specific short period
>> just before the release. If "Release Candidate" has a different meaning now,
>> call it the "Pre-Release Phase", that will end with a "Release Candidate" we
>> can vote on rather confidently. We can't rely on volunteers doing QA on a
>> regular basis, but we can very easily find volunteers willing to stress-test
>> a "nearly final" version in a well defined timeframe (1-2 weeks).
>>
>> I would keep this as a basis for discussing how to structure QA activities
>> in future (after 3.4), if others agree.
>>
>
> This is only a fraction of the picture.  We also need to consider what
> changes are allowed, and how those changes are reviewed, as we
> progress toward a Release Candidate.  That is a critical part of how
> we maintain quality.  If committers are checking in whatever they feel
> like, up to the day we have a RC, then we will have serious quality
> problems.
>
> When we changing code we "invalidate" prior testing.  Not in an
> absolute sense, for most code changes there is an test impact, and
> some portion of the "full test" needs to be redone, to verify both the
> fix, but also that we did not break anything else.  Again, this
> requires discipline from committers.  "Last minute" feature work, or
> even undisciplined bug fixing, can cause more problems than it fixes.
>
> So I think we want the trunk to be fee CTR up to a certain point
> (feature freeze?) and after that point only changes that are
> accompanied by a BZ issue are allowed, but still CTR.  Then at some
> point, after the main test pass is complete, we start prioritizing
> bugs as we create snapshot builds.  We gradually raise the level
> required for bugs to be accepted.  P3, P2, P1.  In the final weeks we
> fix only marked release blockers.   Once we have the RC build we
> branch the code.  At that point (and maybe even earlier) we switch to
> RTC, e.g., no changes are allowed unless they are reviewed first.
>
> Was anything like this done before with OOo?

Yes, more or less in a similar way and I think it is a good way to 
achieve a high quality product.

And on the other hand we should try to provide as much as possible 
automated testing. And volunteers should do manual testing not only RC 
candidates. A continuous testing on dev snapshots would be very much 
appreciated if possible.

Juergen


>
> -Rob
>
>> Regards,
>>   Andrea.


Mime
View raw message