Mailing-List: contact ooo-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: ooo-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of liuyixuan.527@gmail.com
 designates 209.85.214.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAP-ksogc4sX4h=_j9hBKP=HVaRqxEsDZ8k5w_VUr_Sx7mHSVgg@mail.gmail.com>
References: 
 <CAAqNT2tu1=JGGfJfhLu+r2rU=Ottz-y5Zrb46PGC4y+nw8unug@mail.gmail.com>
	<CAJbycq2eMODmZYrMnRE07QepzSmxr1e1RikoqyFRcFaDuZHtvA@mail.gmail.com>
	<CAP-ksogc4sX4h=_j9hBKP=HVaRqxEsDZ8k5w_VUr_Sx7mHSVgg@mail.gmail.com>
Date: Sat, 7 Jul 2012 08:09:26 +0800
Message-ID: 
 <CAAqNT2s=T-qFy_mDrESQq1cG656TW+eaiksFrsZVU956hnfOGA@mail.gmail.com>
Subject: Re: [QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
From: Yi Xuan Liu <liuyixuan.527@gmail.com>
To: ooo-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=14dae93996ed951c7804c43231d2

--14dae93996ed951c7804c43231d2
Content-Type: text/plain; charset=ISO-8859-1

Rob, thanks for your reply. My answer to your question is in below.

1) Not yet. I'll check in code in next week ASAP.
2) It takes about 2 hours to complete the whole text on my W500 laptop. (
CPU:2.53 GHz; Mem: 3GB; OS: XP SP3).
    So, if running 100 times, it would cost about 1 day. Yes, it is also a
good method to check Memory leak :)
3) The order *b *is
4) No, don't restart OpenOffice during the whole PVT.

Also thanks for your advice about the trick mentioned. I'll try it.


On Sat, Jul 7, 2012 at 1:28 AM, Rob Weir <robweir@apache.org> wrote:

> On Thu, Jul 5, 2012 at 11:11 PM, zhangjf <zhangjf@apache.org> wrote:
> > The right url is
> >
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introduction
> >
>
> Great.  Thanks!
>
> A few questions:
>
> 1) Are the scripts and test documents checked into SVN?
>
> 2) How long does it take for a complete run of the tests?
>
> 3) What is the order of the tests?  For example, are you doing:
>
> a) document 1 run 1, document 1 run 2, document 1 run 3...document 1
> run 8, document 2 run 1...document 2 run 8, etc.
>
> or
>
> b) document 1 run 1, document 2 run 1, document 3 run 1...document N
> run 1, then document 1 run 2, document 2 run 2, etc.
>
>
> 4) Within the test do you restart OpenOffice?  If so, do you restart
> after every document?  Or every measurement?
>
>
> If it is at all possible to take more measurements, I think we would
> get more high quality results.  Right now, you take 8 measurements and
> throw away 3 of them (first run, highest time and lowest time).  That
> throws away information and biases the results because the first run
> is probably also slower, so you toss out the two slowest runs but only
> toss out the single fastest run.  But the fastest run is probably also
> the most accurate one, since there are many things in a test that can
> accidentally slow things down, but almost nothing can happen to make a
> test run faster. (assuming the test logic is accurate).
>
> In general, in an experiment, keep all the data you have, and get more
> accurate results by doing more repetitions.
>
> For example, what if we did 100 iterations of each test?  How long
> would that take?
>
> If we did that, it would have some benefits:
>
> 1) We wouldn't need to worry about tossing out high and low values.
> Our error bounds would be good because of the number of runs we have.
>  The impact of any one anomalous measurement will be much smaller.
>
> 2) We could at the same time look at the trend of the measurement over
> the test run.  For example, compare the average of the first 10% of
> the runs with the average of the last 10%.  Is there a difference?  If
> a test slows down over time  that might indicate a memory leak or
> other problem.  You will never find this with only 8 measurements.
>
> 3) It would tell us the distribution of timings, as well as the average.
>
> (Another trick.  If you are going to do N load measurements of the
> same document, maybe start the test run by creating N identical copies
> of the same document on disk.  Then load each copy only once.  That
> helps even out the disk cache and I/O environment compared to loading
> the exact same file N times)
>
> -Rob
>
>
> > On Fri, Jul 6, 2012 at 11:07 AM, Yi Xuan Liu <liuyixuan.527@gmail.com>
> wrote:
> >> Hi, all:
> >>
> >> I wrote a wiki
> >>
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout
> >> PVT project in AOO 3.4.1.
> >> Any comment is welcomed!
>

--14dae93996ed951c7804c43231d2--