incubator-ooo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yi Xuan Liu <liuyixuan....@gmail.com>
Subject Re: [QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
Date Sat, 07 Jul 2012 00:09:26 GMT
Rob, thanks for your reply. My answer to your question is in below.

1) Not yet. I'll check in code in next week ASAP.
2) It takes about 2 hours to complete the whole text on my W500 laptop. (
CPU:2.53 GHz; Mem: 3GB; OS: XP SP3).
    So, if running 100 times, it would cost about 1 day. Yes, it is also a
good method to check Memory leak :)
3) The order *b *is
4) No, don't restart OpenOffice during the whole PVT.

Also thanks for your advice about the trick mentioned. I'll try it.



On Sat, Jul 7, 2012 at 1:28 AM, Rob Weir <robweir@apache.org> wrote:

> On Thu, Jul 5, 2012 at 11:11 PM, zhangjf <zhangjf@apache.org> wrote:
> > The right url is
> >
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introduction
> >
>
> Great.  Thanks!
>
> A few questions:
>
> 1) Are the scripts and test documents checked into SVN?
>
> 2) How long does it take for a complete run of the tests?
>
> 3) What is the order of the tests?  For example, are you doing:
>
> a) document 1 run 1, document 1 run 2, document 1 run 3...document 1
> run 8, document 2 run 1...document 2 run 8, etc.
>
> or
>
> b) document 1 run 1, document 2 run 1, document 3 run 1...document N
> run 1, then document 1 run 2, document 2 run 2, etc.
>
>
> 4) Within the test do you restart OpenOffice?  If so, do you restart
> after every document?  Or every measurement?
>
>
> If it is at all possible to take more measurements, I think we would
> get more high quality results.  Right now, you take 8 measurements and
> throw away 3 of them (first run, highest time and lowest time).  That
> throws away information and biases the results because the first run
> is probably also slower, so you toss out the two slowest runs but only
> toss out the single fastest run.  But the fastest run is probably also
> the most accurate one, since there are many things in a test that can
> accidentally slow things down, but almost nothing can happen to make a
> test run faster. (assuming the test logic is accurate).
>
> In general, in an experiment, keep all the data you have, and get more
> accurate results by doing more repetitions.
>
> For example, what if we did 100 iterations of each test?  How long
> would that take?
>
> If we did that, it would have some benefits:
>
> 1) We wouldn't need to worry about tossing out high and low values.
> Our error bounds would be good because of the number of runs we have.
>  The impact of any one anomalous measurement will be much smaller.
>
> 2) We could at the same time look at the trend of the measurement over
> the test run.  For example, compare the average of the first 10% of
> the runs with the average of the last 10%.  Is there a difference?  If
> a test slows down over time  that might indicate a memory leak or
> other problem.  You will never find this with only 8 measurements.
>
> 3) It would tell us the distribution of timings, as well as the average.
>
> (Another trick.  If you are going to do N load measurements of the
> same document, maybe start the test run by creating N identical copies
> of the same document on disk.  Then load each copy only once.  That
> helps even out the disk cache and I/O environment compared to loading
> the exact same file N times)
>
> -Rob
>
>
> > On Fri, Jul 6, 2012 at 11:07 AM, Yi Xuan Liu <liuyixuan.527@gmail.com>
> wrote:
> >> Hi, all:
> >>
> >> I wrote a wiki
> >>
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout
> >> PVT project in AOO 3.4.1.
> >> Any comment is welcomed!
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message