harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr. <g...@pobox.com>
Subject Re: [testing][cruise-control] how testing infrastructure should be improved
Date Mon, 29 Jan 2007 16:43:06 GMT

On Jan 29, 2007, at 11:26 AM, Vladimir Ivanov wrote:

> On 1/26/07, Geir Magnusson Jr. <geir@pobox.com> wrote:
>>
>>
>> > Created jira should be integrated to the buildtest  
>> infrastructure and
>> > buildtest module should be extended by different application
>> > scenario tests,
>> > reliability tests etc. In general case we will have a long CC cycle
>> > (up to 1
>> > week J ) and results of this runs should be processed with other
>> > procedure
>> > (for example, results should be upload to harmonytest.org and than
>> > jira
>> > issues should be created for all failures).
>>
>> Right - we should probably try to define what we mean by  "short",
>> "medium" and "long" scenarios, and I suggest:
>>
>> 1) what we have now (build and unit test drlvm, classlib) as the
>> "short" cycle for fast failure notification
>>
>> 2) test classlib w/ drlvm and J9  + the iterative test cycle as the
>> "medium" cycle
>>
>> 3) everything you talk about above as the "long" cycle
>
>
> OK. One minor note, some time ago on the dev@ list somebody ask to add
> 'server jit' mode. Is it OK to add it to the current CC ('short'  
> cycle)?

Sure.  Why not.

After staring at my two systems for a couple of days now, I've come  
to the conclusion that our short isn't short enough.  I think we need  
a "shorter" that simply builds stuff.  No tests.

That would be quick "canary in the coalmine" test to help catch  
broken builds, before such breakage affects the longer running  
systems, because right now, the cycle is fairly long just for unit  
tests for drlvm and classlib

geir


>
>>
>> >
>> > It requires some 'standard' interface for all integrated scripts. I
>> > like
>> > classlib interface so how about:
>> >  - call of "ant setup;ant" will run all available scripts;
>> >  - call of "ant -Dmodule=hit setup;ant" will run current version of
>> > CC –
>> > Harmony integration tests;
>> >  - call of "ant -Dmodule=eut setup;ant" will run Eclipse  unit test
>> > etc.
>> >
>>
>> > Note, in this case each module should implement proper 'setup'
>> > target and
>> > has configuration for CC. The root-script will iterate over all
>> > modules to
>> > call their 'setup' and this setup should include whole test setup
>> > (downloading software, adding modules cc-configuration to working
>> > configuration etc).
>> >
>> > Is it OK?
>>
>> I dunno - this sounds like disjoint and separate CC runs, rather than
>> a CC run with multiple projects.
>>
>> For example, I'd like to have a set of "modules", which would be
>> incomplete cc config files, that somehow get glommed into a bigger cc
>> file - maybe the config.xml would have some kind of include  
>> mechanism.
>>
>> Suppose that we have :
>>
>> trunk/cc/
>>     config.xml
>>     modules/
>>        default_module.xml
>>        hut_module.xml
>>        eut_module.xml
>>        iterative_module.xml
>>        dacapo_module.xml
>>        specjbb_module.xml
>>        short_module.xml
>>        medium_module.xml
>>        long_modules.xml
>>
>> so then I could do :
>>
>>   $ ant
>>
>> to get what we have now - runs the default module - or
>>
>>   $ ant -Dmodules=hut,eut,dacapo
>>
>> to run those...
>>
>> Something like "medium_module.xml" would look like  (the following is
>> 'psuedo-code') :
>>
>>     <includeconfig name="default_module.xml"/>
>>     <includeconfig name="dacapo_module.xml"/>
>>
>>
>> so that you can nest this as you want.
>>
>> >
>> > If nobody objects I'll start restructuring of buildtest module and
>> > will try
>> > to integrate one from extensions.
>>
>> Please describe how you want to do it.
>
>
> I think about following option: in the root file we have predefined  
> string.
> Something like 'modules=hut'. In the 'long' mode CC will iterate  
> over all
> entries. The medium cycle depend on users wishes and defined  
> through the
> command line like: 'ant -Dmodules=hut,iterative'. Each module has  
> predefined
> target (for example, 'setup'). In this target script should  
> download all
> software, apply all patches, add module configuration to the  
> current CC
> configuration (just copy content of the module configuration file  
> to the end
> of current configuration) etc. After 'ant <...> setup' we will have
> ready-to-run system with user-defined configuration.
>
>
>> >
>> >
>> > Thanks, Vladimir
>> >
>> >
>> >
>> > PS I think the resulting structure should be easy to extend and may
>> > looks
>> > like this:
>> >
>> > buildtest
>> >
>> > |--config  (default CC configuration to build classlib and DRLVM)
>> >
>> > |--hit (CC configuration to run Harmony classlib&DRLVM tests)
>> >
>> > |--eclipse
>> >
>> >     |-- eut (setup and CC configuration to run eclipse non- 
>> interactive
>> > tests)
>> >
>> >     |-- eclipse3.1.1
>> >
>> >         |-- some scenario
>> >
>> > |-- build.xml (common setup + call of module's 'setup')
>>
>>
>> Interesting.  I think one issue is that it seems like heavy lifting
>> to add a new module - each module becomes a "peer".   What do you
>> think of the other approach above?
>>
>> Either way, we don't want hit, eclipse, etc as peers.  If anything,
>> they should go in a modules/ directory...
>
>
>
> For each module we have at least 2 files: cc-config and build file.  
> But in
> some cases we will have some additional files (patches etc). For
> example, script for testing of JEdit application (issue 3012) has  
> about 15
> files. For me is better to store all staff in one place instead of  
> having
> parallel structure.
>
> thanks, Vladimir
>
> geir
>>
>>


Mime
View raw message