harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject Re: [build-test-infra] Build Test Infrastructure
Date Wed, 26 Jul 2006 13:50:29 GMT

Vladimir Ivanov wrote:
> The more I go further with build&testing infrastructure and think about it
> the more the following overall picture is drawing in my mind.
> Let me share it to see the whole picture and what I see should be next in
> build and testing infrastructure:
> Goal: Harmony quality
> Objectives:
>    - Set-up regular building & build publishing process


>    - Set-up testing process, which:
>        - Provides code integrity (Harmony can be built at each moment of
> time), regressions are seen immediately.

Yep.  We now have a "proprietary" one running at IBM, for which we are
grateful.  The buildtest subproject in Harmony is meant to replace this
at some point and we all will run it.

>        - Done by both at Harmony hosting site and by community members.
>        - Is easy to set-up and run.
>        - Results of testing on community member's platforms can be
> published.
>        - History of test failures, build quality can be tracked.


> Builds.
> Lets consider the following:
> 1.       Regular (daily, weekly?) building of JRE and HDK and tests. 

daily snapshot

> Goal: being regularly published, one can just download and run / test them
> without need to download sources, tools and building Harmony themselves.
> What we need to be built and published? – I suppose we can start with:
>    - Harmony API + DRLVM
>    - Harmony API + J9
>    - Harmony API + JCHVM
>    - API unit tests

I would suggest :

  HDK (classlib + unit tests)
  JDK/JRE (classlib + DRLVM)

and once JCHEVM uses either the classlib or the glue layer is finished,
a JDK/JRE with that.

We won't create anything w/ J9 in it (as that's IBMs proprietary code)
and anyone who wishes to use it can take the HDK and drop J9 right in
very easily.

That said, I do think we still want people to be independently running
(as part fo the test loop) Classlib + J9 because testing with the broad
variety of VMs available to us will is useful.

> Do we want to provide capabilities for people to install "build tool" and
> have Harmony built regularly on their platforms? – I guess yes.

Yes - that is the point of the buildtest/ project.

> ****I can (and plan) do a simple ant script for building and archiving
> binaries (If no objection).

We already have a script for building the HDK and JRE.  The publishing
part should be fairly straightforward.  I imagine we'll have volunteers
that produce the designated nightly snapshots for a given platform so we
can get the widest variety of OS/chipset as possible.

> Once it is done, I can start publishing the binaries more or less regularly
> at some site and provide links at our wiki pages – is it worth doing?

We would put them in the snapshot directory that we use now.


> 2.       We establish continuous building to make sure we have consistent
> workspace. Cruise Control is OK for this purpose.
> What we build:
>    - Harmony API
>    - DRLVM
>    - JCHVM
> Do we want to provide capabilities for people to install and have "Cruise
> Control – based tool" running on their platforms? – I guess yes (correct me
> if I'm wrong), in case if somebody is interested in keeping Harmony
> buildable and runnable on his favorite platform (FreeBSD, for example).

That's what the buildtest subproject is for.  Have you looked at it?

> Testing.
> I would propose to consider the following levels of testing.
> 1.       Pre-integration testing – quickly executed set of tests to be run
> to make sure a patch does not worsen Harmony quality.
> Goal: for a community member or committer to check that the patch does not
> worsen Harmony quality.
> ****This is already discussed, agreed and implemented for both API and VM.
> Do we need something more to be done here?

I hope not.

> 2.       Code integrity testing – done by regular often building of Harmony
> components by Cruise Control.
> Goal: to make sure no "compilation errors" were introduced by recent
> commits, if introduced, then, notification along with list of recent
> commits
> is sent to all.
> ****This is what is done recently by Geir by Cruise Control script.

Yes, I am doing it, but the goal isn't to have only one person doing it.
 The goal of the buildtest/ subproject


is to help *anyone* that wants to help to easily checkout the system,
have it auto-setup as much as possible, and then just run.

What remains to be worked out is the email and notification aggregation,
so that we can have a "dashboard" or matrix of all the platforms being
tested, with their current status and history.  This is an important
thing to get done, and I'm hoping someone volunteers.

> 3.       Smoke testing – a number of tests should be run over received via
> Cruise Control build.
> Goal: to make sure no regressions were introduced by recent commits, the
> built runtime can do its basic operations.
> If regression is introduced, then, notification along with list of recent
> commits is sent to all.
> ****This is what is done recently by Geir in Cruise Control script.


> 4.       Full testing - the whole set of Harmony tests are run on regular
> builds, results are published.
> Goal: to see what happened with Harmony quality on the whole set of
> automated tests, see new bugs, see quality of Harmony runtimes.
> Users should be able to do this kind of testing on their specific platforms
> and publish results on Harmony web site.
> ****This script (prototype) I implemented, see issue 984.

I thought that the classlib tests are run as well w/ the CC script right
now, but will check.

> 5.       Application testing – people enable applications, try already
> enabled applications on fresh builds, publish results.
> Goal: to see what happened with Harmony quality on various applications
> people like.
> We can start from having a wiki page (which partially exists in some form)
> on which community publishes results of running applications on various
> Harmony builds.
> ****Do we need here more?

Maybe eventually we'll want to automate.  We certainly want to get Gump
runs going too for this.

> 6.       [optional] Reliability testing – either special reliability tests
> or some applications are executed during long time (from one regular build
> to another).
> Goal: to make sure we have reliable, stable, long running runtime.
> For future.
> thanks, Vladimir

Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org

View raw message