incubator-ooo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Weir <robw...@apache.org>
Subject Re: [QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
Date Fri, 06 Jul 2012 17:28:04 GMT
On Thu, Jul 5, 2012 at 11:11 PM, zhangjf <zhangjf@apache.org> wrote:
> The right url is
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introduction
>

Great.  Thanks!

A few questions:

1) Are the scripts and test documents checked into SVN?

2) How long does it take for a complete run of the tests?

3) What is the order of the tests?  For example, are you doing:

a) document 1 run 1, document 1 run 2, document 1 run 3...document 1
run 8, document 2 run 1...document 2 run 8, etc.

or

b) document 1 run 1, document 2 run 1, document 3 run 1...document N
run 1, then document 1 run 2, document 2 run 2, etc.


4) Within the test do you restart OpenOffice?  If so, do you restart
after every document?  Or every measurement?


If it is at all possible to take more measurements, I think we would
get more high quality results.  Right now, you take 8 measurements and
throw away 3 of them (first run, highest time and lowest time).  That
throws away information and biases the results because the first run
is probably also slower, so you toss out the two slowest runs but only
toss out the single fastest run.  But the fastest run is probably also
the most accurate one, since there are many things in a test that can
accidentally slow things down, but almost nothing can happen to make a
test run faster. (assuming the test logic is accurate).

In general, in an experiment, keep all the data you have, and get more
accurate results by doing more repetitions.

For example, what if we did 100 iterations of each test?  How long
would that take?

If we did that, it would have some benefits:

1) We wouldn't need to worry about tossing out high and low values.
Our error bounds would be good because of the number of runs we have.
 The impact of any one anomalous measurement will be much smaller.

2) We could at the same time look at the trend of the measurement over
the test run.  For example, compare the average of the first 10% of
the runs with the average of the last 10%.  Is there a difference?  If
a test slows down over time  that might indicate a memory leak or
other problem.  You will never find this with only 8 measurements.

3) It would tell us the distribution of timings, as well as the average.

(Another trick.  If you are going to do N load measurements of the
same document, maybe start the test run by creating N identical copies
of the same document on disk.  Then load each copy only once.  That
helps even out the disk cache and I/O environment compared to loading
the exact same file N times)

-Rob


> On Fri, Jul 6, 2012 at 11:07 AM, Yi Xuan Liu <liuyixuan.527@gmail.com> wrote:
>> Hi, all:
>>
>> I wrote a wiki
>> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout
>> PVT project in AOO 3.4.1.
>> Any comment is welcomed!

Mime
View raw message