harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [testing][cruise-control] how testing infrastructure should be improved
Date Fri, 26 Jan 2007 12:55:38 GMT

On Jan 26, 2007, at 6:50 AM, Vladimir Ivanov wrote:

> Hello everyone,
>
> while we have more or less stable CC it is time to do work to  
> extend Harmony
> testing :)
> Yes, I know that we have some gap in stability on x86_64 platform  
> and on Red
> Hat Linux so CC runs tests in the 'perTest' mode but anyway I want to
> summarize our discussions about testing and continue extend testing
> capability.
>
>
>
> The current buildtest infrastructure includes only one case for  
> testing:
> cyclic run of Harmony pre-integration tests. Also we have jira  
> issues with
> iterative runs, eclipse "hello world" and JEDIT scenarios. And  
> community
> members time to time runs applications and report their execution  
> status.
>
>
>
> The idea was to collect all test-run scripts into buildtest  
> directory and
> implement configurable CC.

Yep!

>
> Created jira should be integrated to the buildtest infrastructure and
> buildtest module should be extended by different application  
> scenario tests,
> reliability tests etc. In general case we will have a long CC cycle  
> (up to 1
> week J ) and results of this runs should be processed with other  
> procedure
> (for example, results should be upload to harmonytest.org and than  
> jira
> issues should be created for all failures).

Right - we should probably try to define what we mean by  "short",  
"medium" and "long" scenarios, and I suggest:

1) what we have now (build and unit test drlvm, classlib) as the  
"short" cycle for fast failure notification

2) test classlib w/ drlvm and J9  + the iterative test cycle as the  
"medium" cycle

3) everything you talk about above as the "long" cycle


>
>
> It requires some 'standard' interface for all integrated scripts. I  
> like
> classlib interface so how about:
>  - call of "ant setup;ant" will run all available scripts;
>  - call of "ant -Dmodule=hit setup;ant" will run current version of  
> CC –
> Harmony integration tests;
>  - call of "ant -Dmodule=eut setup;ant" will run Eclipse  unit test  
> etc.
>

> Note, in this case each module should implement proper 'setup'  
> target and
> has configuration for CC. The root-script will iterate over all  
> modules to
> call their 'setup' and this setup should include whole test setup
> (downloading software, adding modules cc-configuration to working
> configuration etc).
>
> Is it OK?

I dunno - this sounds like disjoint and separate CC runs, rather than  
a CC run with multiple projects.

For example, I'd like to have a set of "modules", which would be  
incomplete cc config files, that somehow get glommed into a bigger cc  
file - maybe the config.xml would have some kind of include mechanism.

Suppose that we have :

  trunk/cc/
     config.xml
     modules/
        default_module.xml
        hut_module.xml
        eut_module.xml
        iterative_module.xml
        dacapo_module.xml
        specjbb_module.xml
        short_module.xml
        medium_module.xml
        long_modules.xml

so then I could do :

   $ ant

to get what we have now - runs the default module - or

   $ ant -Dmodules=hut,eut,dacapo

to run those...

Something like "medium_module.xml" would look like  (the following is  
'psuedo-code') :

     <includeconfig name="default_module.xml"/>
     <includeconfig name="dacapo_module.xml"/>


so that you can nest this as you want.

>
> If nobody objects I'll start restructuring of buildtest module and  
> will try
> to integrate one from extensions.

Please describe how you want to do it.

>
>
> Thanks, Vladimir
>
>
>
> PS I think the resulting structure should be easy to extend and may  
> looks
> like this:
>
> buildtest
>
> |--config  (default CC configuration to build classlib and DRLVM)
>
> |--hit (CC configuration to run Harmony classlib&DRLVM tests)
>
> |--eclipse
>
>     |-- eut (setup and CC configuration to run eclipse non-interactive
> tests)
>
>     |-- eclipse3.1.1
>
>         |-- some scenario
>
> |-- build.xml (common setup + call of module's 'setup')


Interesting.  I think one issue is that it seems like heavy lifting  
to add a new module - each module becomes a "peer".   What do you  
think of the other approach above?

Either way, we don't want hit, eclipse, etc as peers.  If anything,  
they should go in a modules/ directory...

geir


Mime
View raw message