harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Fedotov" <alexei.fedo...@gmail.com>
Subject Re: [buildtest][harness][p-unit] performance measurement infrastructure
Date Tue, 26 Jun 2007 12:18:50 GMT
Andrew,

I haven't tried demo yet. I've noticed that you added a test for new
functionality and decided to try it. I looked for
trunk/punit.test/README.txt, then tried to find test targets in
build.xml files. Finally I've tried Eclipse test runner:

1. Started Eclipse with the workspace set to trunk/
2. Added punit, punit.extension, punit.samples, punit.test to the project tree.
3. Built punit and punit.extension by means of Ant.
4. Pressed right mouse button over punit.test and selected Run as -> JUnit test
5. Among 113 testcases 6 testcases testcases of
tests.api.org.punit.testclasses.JUnitAnnotationTestClass reported an
error.
6. Also I see that tests.api.org.punit.testclasses.LoopTestClass was
absent in the test run results.

When I ran both problematic classes as standalone applications I could
see that testcases pass. That's make me think that the test design is
ok, but it is not aligned with what Eclipse expects.

To summarize, I see errors and missed tests. This might be a bug, or
just I don't know enough details on how to run all tests.




On 6/26/07, Andrew Zhang <zhanghuangzhu@gmail.com> wrote:
> On 6/26/07, Alexei Fedotov <alexei.fedotov@gmail.com> wrote:
> >
> > Andrew,
> >
> > That's great. You are really quick. I was still thinking when your
> > solution came up. I was able to build the project and run tests one by
> > one, though I wasn't able to use the perfect Eclipse plugin for JUnit
> > tests. Probably it could make sense to use just an ordinary junit for
> > testing p-unit itself.
>
>
> Hi Alexei,
>
> Do you mean p-unit tests here? p-unit uses both ordinary junit 4 and p-unit
> itself to run all tests for p-unit. They're AllTests(JUnit test suite
> format) and PUnitAllTests(p-unit test suite format) under
> punit.test\src\tests\api\org\punit\all directory.
> Files in punit.sample are run one by one, because each file is supposed to
> demo one functionality. :) It's for demo (not testing).
>
> Thanks!
>
> <snip>
> > On 6/25/07, Andrew Zhang <zhanghuangzhu@gmail.com> wrote:
> > > On 6/22/07, Alexei Fedotov <alexei.fedotov@gmail.com> wrote:
> > > >
> > > > Andrew,
> > > > Thanks for your explanations.
> > > >
> > > > Let me address the main problem I see to migrate a stress test suite
> > > > to p-unit. To which extend may I count on your support at further
> > > > p-unit development? I listed very specific functionality which
> > > > probably didn't suite well your original idea.
> > >
> > >
> > > Hi Alexei,
> > >
> > > Would you please check the latest sample code samples.LoopTestSample in
> > > punit.sample project? Is it the same as the original idea of toStop,
> > > toAbort? Thanks!
> > >
> > > On 6/22/07, Andrew Zhang <zhanghuangzhu@gmail.com> wrote:
> > > > > On 6/22/07, Alexei Fedotov <alexei.fedotov@gmail.com> wrote:
> > > > > >
> > > > > > Andrew,
> > > > > > This is very interesting. Thank you for the reference.
> > > > >
> > > > >
> > > > > Hi Alexei,  please see my comments inline
> > > > >
> > > > > I've got few questions trying to understand your framework.
> > > > > > 1. Why an interface Watcher extends Serializable?
> > > > >
> > > > >
> > > > > Because it's for performance comparison between different vms. The
> > > > runner
> > > > > will be serialized, so that the vm can load the right runner. You
> > can
> > > > > consider it like serializing the configuration for the vm.
> > > > >
> > > > > 2. Is there any project which uses p-unit for testing? Why have you
> > > > started
> > > > > > it?
> > > > >
> > > > >
> > > > > Polepos (https://sourceforge.net/projects/polepos) is going to
> > refactor
> > > > its
> > > > > code to use p-unit.
> > > > >
> > > > > And I know serveral community members use p-unit to test concurrency
> > > > > performance(by running the existing JUnit tests).
> > > > >
> > > > > I started it because I found that I can't do something as I want
> > with
> > > > JUnit,
> > > > > :) like concurrency, performance test. Why do I have to start
> > thread,
> > > > and
> > > > > join, and check the result at the end of join everytime? why do I
> > need
> > > > to
> > > > > use System.currentMilliseconds and sysout... :)  So I wrote p-unit.
> > > > >
> > > > > 3. Running tests in several VM is great, but how could I compare
> > these
> > > > > > results?
> > > > >
> > > > >
> > > > > There're serveral build-in reports in p-unit. The most
> > straightforward
> > > > one
> > > > > is pdf format, which contains performance chart. The results of
> > > > different vm
> > > > > for the same method are shown in the same page, so that you can know
> > > > which
> > > > > one is better easily. It's also easy to plugin your own report by
> > > > > implementing  PUnitEventListener.
> > > > >
> > > > > 4. If I need to increase a parameter until the test fails and then
> > > > > > report the final value as a test performance metrics how this
> > should
> > > > > > be done in p-unit?
> > > > >
> > > > >
> > > > > hm... this functionality is not supported in p-unit....
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On 6/22/07, Andrew Zhang <zhanghuangzhu@gmail.com> wrote:
> > > > > > > On 6/21/07, Alexei Fedotov <alexei.fedotov@gmail.com>
wrote:
> > > > > > > >
> > > > > > > > Hello Vladimir,
> > > > > > > >
> > > > > > > > Could yout please give few hints how one should write
> > performance
> > > > > > > > tests using the harness? I found an option "-performance"
but
> > > > cannot
> > > > > > > > see anything else. I wonder how a testcase could report
a
> > > > performance
> > > > > > > > metrics.
> > > > > > > >
> > > > > > > > Actually I'm thinking now of reviving comparative
stress
> > testing
> > > > > > approach.
> > > > > > >
> > > > > > >
> > > > > > > Hi Alexei,
> > > > > > >
> > > > > > > Do you have a chance to look at p-unit, (
> > > > http://p-unit.sourceforge.net/
> > > > > > )?
> > > > > > >
> > > > > > > I think it fulfills most of the requests listed in the
> > documents.
> > > > > > >
> > > > > > > Another advantage of p-unit is that it can compare the
> > performance
> > > > > > between
> > > > > > > different vms.
> > > > > > >
> > > > > > > One problem may be the license (GPL now, going to change
the
> > license
> > > > to
> > > > > > GPL
> > > > > > > v3). If we really take it serious, I can relicense it
> > immediately.
> > > > > > >
> > > > > > > [1]
> > > > > > > >
> > > > > >
> > > >
> > http://mail-archives.apache.org/mod_mbox/harmony-dev/200605.mbox/%3C6694B22B6436BC43B429958787E4549802012354@mssmsx402nb%3E
> > > > > > > >
> > > > > > > > --
> > > > > > > > With best regards,
> > > > > > > > Alexei,
> > > > > > > > ESSD, Intel
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Andrew Zhang
> > > > > > >
> > > > > > > http://zhanghuangzhu.blogspot.com/
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > With best regards,
> > > > > > Alexei,
> > > > > > ESSD, Intel
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Andrew Zhang
> > > > >
> > > > > http://zhanghuangzhu.blogspot.com/
> > > > >
> > > >
> > > >
> > > > --
> > > > With best regards,
> > > > Alexei,
> > > > ESSD, Intel
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew Zhang
> > >
> > > http://zhanghuangzhu.blogspot.com/
> > >
> >
> >
> > --
> > With best regards,
> > Alexei,
> > ESSD, Intel
> >
>
>
>
> --
> Best regards,
> Andrew Zhang
>
> http://zhanghuangzhu.blogspot.com/
>


-- 
With best regards,
Alexei,
ESSD, Intel

Mime
View raw message