harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Beyer" <nbe...@gmail.com>
Subject Re: [build-and-test] Thoughts about testing infra
Date Wed, 05 Mar 2008 22:31:16 GMT
On Wed, Mar 5, 2008 at 11:13 AM, Tim Ellison <t.p.ellison@gmail.com> wrote:
> Stepan Mishura wrote:
>  > Hi folks,
>  >
>  > I've moved 2.0 build-and-test's branch to the trunk - the branch was
>  > used during M2-M5 testing and IMO it proved to be very useful (also I
>  > tagged the former trunk as 1.0.)
>
>  Hurray!
>
>
>
>  > I think that this is a good time to collect ideas and hear opinions
>  > what should be improved and possible directions for test infra
>  > development.
>  >
>  > I have to admit that the current testing infra is far from being
>  > perfect – there a lot for work to make is more usable, cleaning up
>  > current scripts, sort out all know issues with tests/scenarios and so
>  > on. But I'd like to address problems I believe of different level. I
>  > have an impression that Harmony tests suites are still in shadow (IOW,
>  > IMHO they are undervalued) of testing automation (i.e. CruiseControl
>  > and support scripts) – and most of folks prefer to contribute to the
>  > code development. And I believe that it is possible to make tests
>  > development also interesting too. IMO the missed point that most of
>  > Harmony test suites can be used as stand alone suites to verify/test
>  > different implementations – in other words, the suites are not tightly
>  > coupled with DRL VM, for example, it is possible to run them against
>  > IBM VME (BTW, may it be makes sense to run full set of suites against
>  > IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
>  >
>  > I'm going to update the test infra home page [1] with info related to
>  > the renewed trunk. Also I think the page should contain some words
>  > about project's testing strategy: i.e. how Harmony QA and release
>  > process are organized - integrity/snapshot testing, milestones, code
>  > freeze and etc., what suites are there (their description and
>  > purpose). I think this may encourage people to invest efforts into
>  > developing tests.
>  >
>  > Other ideas and thoughts?
>
>  Thanks for your investment in the test suites, and taking time to think
>  about the improvements.
>
>  The thing I struggle with is understanding the current system well
>  enough to make a valuable contribution.
>
>  It would be really good to have a newbies' guide to the test suites
>  document on the website with step-by-step instructions (like the quick
>  help for contributors document).

I concur with sentiment. I don't get a chance to continually work with
the tool, so I start fresh every time I come back to and it seems
every time I have to stumble over a bunch of little problems for about
a day before getting it working again. I don't know if it's been
resolved, but the issues with running on Ant 1.7 instead of Ant 1.6.5
we always painful.

-Nathan

>
>  While the architecture documentation would be interesting, I think many
>  people work by running the simple case, then get dragged into
>  investigating the complex cases when their needs require it.
>
>  Regards,
>  Tim
>
>
>
>  > [1] http://harmony.apache.org/subcomponents/buildtest/index.html
>  >
>  > Thanks,
>  > Stepan.
>  >
>
Mime
View raw message