harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [buildtest] moving the testing infrastructure forward
Date Tue, 21 Nov 2006 03:14:36 GMT
[btw - you sent this to harmony-dev@incubator....]

Stefano Mazzocchi wrote:
> The basic idea is to have a testing framework of 'builders' that just do
> the dirty work and send the results to a centralized location that does
> analysis and presentation.

Yes.

> 
> We currently have a seed of this at
> 
>  https://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk/
> 
> The theory is that you do
> 
>  > svn co https://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk/
>  > ant setup (this gets all the stuff you need and compiles everything)
>  > ant (this starts cruisecontrol)
> 
> you can do it on your own machine, you can run it continuously or simple
> one every other day, you can run it virtualized or not, doesn't matter.
> 
> The problem is that the practice doesn't meet the theory, so let's try
> to fix what's wrong with the current implementation:
> 
> 1) the config-full.xml is OS independent but it's not ARCH independent
> (it's currently frozen to i32). This needs to be fixed.
> 
> 2) cruisecontrol starting scripts are really ugly and the README
> suggests that you patch them before you start. This should be done
> automatically during 'ant setup' using a <replace> task (or by providing
> our own startup scripts for CC which is not that hard)

That's fine.

> 
> 3) if we allow every single CC instance to send email to this list we'll
> be flooded. I think we should turn that off and implement our own
> 'publishing' plugin for CC that publishes the results to harmonytest.org
> (which should redirect to harmony.zone.apache.org as soon as we can). It
> should be the receiving end to send email to the list in case shit
> happens (providing a summary of the results).

It never was intended that every instance out there was sending mail to 
harmony dev.  Nor do I think that every instance out there should be 
sending to anywhere via a plugin.

What I thought would be good, an I sent mail re this over the weekend, 
was to fill out a "matrix" of platforms and servers, and have only one 
per architecture/OS doing the work.

http://wiki.apache.org/harmony/Automated_Testing

We could certainly use uploads to wherever, but having each one notify 
the list on state change (build broke... build fixed...) is important as 
an event to alert the community.  Much better than polling harmony-whatever.

Of course, we could have the centralized thingy sending mail on 
breakage, but that's essentially a refinement, rather than a change.


> 
> 4) we should *NOT* add things like japitools or findbugs to this testing
> environment as only things that are 'OS/ARCH' dependent are useful in
> this distributed testing. Anything that has to do with static analysis
> can be performed in a centralized location more efficiently.

Agreed - that's the intent - but have it in the same overall framework 
if possible.

So we have one machine in the community - could be the central one, 
could be a volunteer - that runs those targets for CC, while everyone 
that's signed up to be a OS/ARCH tester doesn't.

But at any time, any member of the community can checkout the framework 
that everyone else is using, and just choose to run those same 
japitools/* targets if they want.

> 
> 5) config-full.xml currently runs three targets "classlib, drlvm and
> drlvm-test" where classlib runs the build and runs the tests. I think
> classlib should be split in three: java, native and tests. I also note
> that classlib tests need a JVM, so we need to come up with a reasonable
> strategy for all that.

Yes - each should actually be broken up into sequences :

1)  classlib -> update/build and then unit tests using j9 (IMO) to test 
classlib in isolation of changes from DRLVM.  It used to be that way IIRC.

2) drlvm - > update/build and then unit tests.

3) classlib unit tests using DRLVM

4) drlvm + classlib running app tests - tomcat tests, the derby suite, 
whatever...  basically strategically chosen user apps.

geir


Mime
View raw message