harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stepan Mishura" <stepan.mish...@gmail.com>
Subject Re: [build-and-test] Thoughts about testing infra
Date Thu, 06 Mar 2008 08:35:06 GMT
On 3/5/08, Tim Ellison <t.p.ellison@gmail.com> wrote:
> Stepan Mishura wrote:
> > Hi folks,
> >
> > I've moved 2.0 build-and-test's branch to the trunk - the branch was
> > used during M2-M5 testing and IMO it proved to be very useful (also I
> > tagged the former trunk as 1.0.)
>
> Hurray!
>
> > I think that this is a good time to collect ideas and hear opinions
> > what should be improved and possible directions for test infra
> > development.
> >
> > I have to admit that the current testing infra is far from being
> > perfect – there a lot for work to make is more usable, cleaning up
> > current scripts, sort out all know issues with tests/scenarios and so
> > on. But I'd like to address problems I believe of different level. I
> > have an impression that Harmony tests suites are still in shadow (IOW,
> > IMHO they are undervalued) of testing automation (i.e. CruiseControl
> > and support scripts) – and most of folks prefer to contribute to the
> > code development. And I believe that it is possible to make tests
> > development also interesting too. IMO the missed point that most of
> > Harmony test suites can be used as stand alone suites to verify/test
> > different implementations – in other words, the suites are not tightly
> > coupled with DRL VM, for example, it is possible to run them against
> > IBM VME (BTW, may it be makes sense to run full set of suites against
> > IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
> >
> > I'm going to update the test infra home page [1] with info related to
> > the renewed trunk. Also I think the page should contain some words
> > about project's testing strategy: i.e. how Harmony QA and release
> > process are organized - integrity/snapshot testing, milestones, code
> > freeze and etc., what suites are there (their description and
> > purpose). I think this may encourage people to invest efforts into
> > developing tests.
> >
> > Other ideas and thoughts?
>
> Thanks for your investment in the test suites, and taking time to think
> about the improvements.
>
> The thing I struggle with is understanding the current system well
> enough to make a valuable contribution.
>

Yes, to automate the suite runs you have to study how a whole system
(i.e. BTI) works.

And my point was that you don't need to understand BTI to contribute
to a particular suite. You should only grab the suite and start
running it. This is true for classlib unit tests, drlvm suites ... ,
isn't it? But the same applies for the rest of them: Geronimo unit
tests, Eclipse unit tests, functional suite, VTS VM suite and so on.

> It would be really good to have a newbies' guide to the test suites
> document on the website with step-by-step instructions (like the quick
> help for contributors document).
>

Sure, I'm going to improve the docs.

Thanks,
Stepan.

> While the architecture documentation would be interesting, I think many
> people work by running the simple case, then get dragged into
> investigating the complex cases when their needs require it.
>
> Regards,
> Tim
>
> > [1] http://harmony.apache.org/subcomponents/buildtest/index.html
> >
> > Thanks,
> > Stepan.
> >
>

Mime
View raw message