harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [testing][cruise-control] how testing infrastructure should be improved
Date Tue, 30 Jan 2007 06:46:58 GMT

On Jan 30, 2007, at 5:28 AM, Vladimir Ivanov wrote:

> On 1/29/07, Geir Magnusson Jr. <geir@pobox.com> wrote:
>>
>>
>> > OK. One minor note, some time ago on the dev@ list somebody ask  
>> to add
>> > 'server jit' mode. Is it OK to add it to the current CC ('short'
>> > cycle)?
>>
>> Sure.  Why not.
>>
>> After staring at my two systems for a couple of days now, I've come
>> to the conclusion that our short isn't short enough.  I think we need
>> a "shorter" that simply builds stuff.  No tests.
>>
>> That would be quick "canary in the coalmine" test to help catch
>> broken builds, before such breakage affects the longer running
>> systems, because right now, the cycle is fairly long just for unit
>> tests for drlvm and classlib
>>
>> geir
>
>
> My timings are:
> WinXP: classlib ~17 min, drlvm ~10, cl_tests ~34, drlvm_tests ~55
> Linux ia32: ~15, ~11, ~40, ~50min
> Linux em64t: ~11, ~11, ~120, ~50min
>
> So, the cycle duration is about 2hours (I hope we can switch Linux  
> em64t
> from 'perTest' to 'once' mode soon :)).

I'm running x86_64 (remember, we decided to use that token rather  
than em64t :) in 'once' mode - I didn't ahve to switch to perTest.

> If we define the 'short' cycle as 'build the classlib and the  
> drlvm' the
> minimal cycle will be reduced up to ~30 min. In this case we will have
> 'short' cycle ~30 min, user defined 'medium' cycle and all-included  
> 'long'.
> Is it OK?

I think so.  I think we need better names that are descriptive rather  
than relative, so we can add more w/o resorting to things like "ultra  
medium short long" or whatever...

so maybe :

"build-profile" : just build classlib, drlvm and tools
"unit_profile" : runs the drlvm tests and classlib tests, can be used  
after "build-profile"
"interative_profile" : can run after either the above, does iterative  
testing
"functional1_profile" : ...
etc

?

I don't know CC well enough - can I have these pre-set things, and  
tell CC to do "build-profile" only, or do build-profile, and then if  
that passes, do the unit-profile?

geir

>
> thanks, Vladimir
>
>
>
>> >
>> >>
>> >> >
>> >> > It requires some 'standard' interface for all integrated  
>> scripts. I
>> >> > like
>> >> > classlib interface so how about:
>> >> >  - call of "ant setup;ant" will run all available scripts;
>> >> >  - call of "ant -Dmodule=hit setup;ant" will run current  
>> version of
>> >> > CC –
>> >> > Harmony integration tests;
>> >> >  - call of "ant -Dmodule=eut setup;ant" will run Eclipse   
>> unit test
>> >> > etc.
>> >> >
>> >>
>> >> > Note, in this case each module should implement proper 'setup'
>> >> > target and
>> >> > has configuration for CC. The root-script will iterate over all
>> >> > modules to
>> >> > call their 'setup' and this setup should include whole test  
>> setup
>> >> > (downloading software, adding modules cc-configuration to  
>> working
>> >> > configuration etc).
>> >> >
>> >> > Is it OK?
>> >>
>> >> I dunno - this sounds like disjoint and separate CC runs,  
>> rather than
>> >> a CC run with multiple projects.
>> >>
>> >> For example, I'd like to have a set of "modules", which would be
>> >> incomplete cc config files, that somehow get glommed into a  
>> bigger cc
>> >> file - maybe the config.xml would have some kind of include
>> >> mechanism.
>> >>
>> >> Suppose that we have :
>> >>
>> >> trunk/cc/
>> >>     config.xml
>> >>     modules/
>> >>        default_module.xml
>> >>        hut_module.xml
>> >>        eut_module.xml
>> >>        iterative_module.xml
>> >>        dacapo_module.xml
>> >>        specjbb_module.xml
>> >>        short_module.xml
>> >>        medium_module.xml
>> >>        long_modules.xml
>> >>
>> >> so then I could do :
>> >>
>> >>   $ ant
>> >>
>> >> to get what we have now - runs the default module - or
>> >>
>> >>   $ ant -Dmodules=hut,eut,dacapo
>> >>
>> >> to run those...
>> >>
>> >> Something like "medium_module.xml" would look like  (the  
>> following is
>> >> 'psuedo-code') :
>> >>
>> >>     <includeconfig name="default_module.xml"/>
>> >>     <includeconfig name="dacapo_module.xml"/>
>> >>
>> >>
>> >> so that you can nest this as you want.
>> >>
>> >> >
>> >> > If nobody objects I'll start restructuring of buildtest  
>> module and
>> >> > will try
>> >> > to integrate one from extensions.
>> >>
>> >> Please describe how you want to do it.
>> >
>> >
>> > I think about following option: in the root file we have predefined
>> > string.
>> > Something like 'modules=hut'. In the 'long' mode CC will iterate
>> > over all
>> > entries. The medium cycle depend on users wishes and defined
>> > through the
>> > command line like: 'ant -Dmodules=hut,iterative'. Each module has
>> > predefined
>> > target (for example, 'setup'). In this target script should
>> > download all
>> > software, apply all patches, add module configuration to the
>> > current CC
>> > configuration (just copy content of the module configuration file
>> > to the end
>> > of current configuration) etc. After 'ant <...> setup' we will have
>> > ready-to-run system with user-defined configuration.
>> >
>> >
>> >> >
>> >> >
>> >> > Thanks, Vladimir
>> >> >
>> >> >
>> >> >
>> >> > PS I think the resulting structure should be easy to extend  
>> and may
>> >> > looks
>> >> > like this:
>> >> >
>> >> > buildtest
>> >> >
>> >> > |--config  (default CC configuration to build classlib and  
>> DRLVM)
>> >> >
>> >> > |--hit (CC configuration to run Harmony classlib&DRLVM tests)
>> >> >
>> >> > |--eclipse
>> >> >
>> >> >     |-- eut (setup and CC configuration to run eclipse non-
>> >> interactive
>> >> > tests)
>> >> >
>> >> >     |-- eclipse3.1.1
>> >> >
>> >> >         |-- some scenario
>> >> >
>> >> > |-- build.xml (common setup + call of module's 'setup')
>> >>
>> >>
>> >> Interesting.  I think one issue is that it seems like heavy  
>> lifting
>> >> to add a new module - each module becomes a "peer".   What do you
>> >> think of the other approach above?
>> >>
>> >> Either way, we don't want hit, eclipse, etc as peers.  If  
>> anything,
>> >> they should go in a modules/ directory...
>> >
>> >
>> >
>> > For each module we have at least 2 files: cc-config and build file.
>> > But in
>> > some cases we will have some additional files (patches etc). For
>> > example, script for testing of JEdit application (issue 3012) has
>> > about 15
>> > files. For me is better to store all staff in one place instead of
>> > having
>> > parallel structure.
>> >
>> > thanks, Vladimir
>> >
>> > geir
>> >>
>> >>
>>
>>


Mime
View raw message