harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Mazzocchi <stef...@apache.org>
Subject Re: [drlvm][unit] 100% of class library tests pass
Date Thu, 16 Nov 2006 16:58:17 GMT
Tim Ellison wrote:
> Pavel Ozhdikhin wrote:
>> We have to evolving systems - classlib and DRLVM. To check commits to
>> classlib we need a stable DRLVM which can pass 100% of HUT. Otherwise it's
>> impossible to use DRLVM for pre-commit testing - you never know whether
>> your
>> test fail because of your patch or due to latest changes in DRLVM.
>>
>> I remember the time when DRLVM and Jitrino actively evolved - for some time
>> JIT had to use an older version of DRLVM which could pass all commit
>> criteria because newer versions suffered from regressions. And finally we
>> came to comon strict commit criteria which prevented regressions in both VM
>> and JIT.
>>
>> To avoid regressions using DRLVM in classlib testing I see 3 possible
>> solutions:
>>
>> 1. Use one fixed DRLVM version which can pass 100% HUT test. Update this
>> version from time to time.
>>    Pros: + Less time to run DRLVM pre-commit tests
>>              + Classlib does not suffer from regressions in DRLVM
>>    Cons: - DRLVM will suffer from regressions
>>               - Classlib can not use the latest DRLVM
>>               - Need additional efforts to regain stability on DRLVM
>>                 when we want to update the version for classlib testing
>>
>> 2. Add HUT to CruiseControl testing on DRLVM and rollback commits causing
>> regressions
>>    Pros: + Less time to run DRLVM pre-commit tests
>>              + Classlib can use the latest DRLVM
>>    Cons: - Classlib can suffer from DRLVM regressions (time lag before
>> rollback)
>>               - It is not always clear which commit caused a regression
>>               - Rollbacks are costly
>>
>> 3. Add HUT to the commit criteria for DRLVM
>>     Pros: + Classlib always can use the latest DRLVM
>>               + DRLVM has no regressions regarding to HUT
>>     Cons: - More time to run DRLVM pre-commit tests (I was told that HUT
>>                   take 25 minutes running in single JVM mode)
>>
>> I think that preventing a problem is better than solving it afterwards. So,
>> I personally would choose the 3rd approach, don't mind against the second
>> and dislike the first one. Probably some combination of these is possible.
> 
> While I appreciate the desire to keep things stable, I think it is
> unreasonable to ask developers to run the entire test suite each time.
> As we have seen in the classlib code, running targeted tests before
> commit and leaving the build machines to run continuous tests ensures
> that we are productive and are notified of breakages in time to easily
> back out a patch and re-evaluate.
> 
> With the amount of machine time we have running harmony tests on
> different cpu's/os's/compilers/etc we are getting better coverage than
> any individual could be expected to provide.
> 
> Which is a long way of saying I think option (2) above is best -- and
> relies on the bid machines letting us know if things break, and the
> commitment from all of us to go straight in and fix it.

Agreed, "commit then review" scales better.

There is no way a single developer running tests on her/his own machines
can know if it's safe to commit a patch... we are not making a release
for every commit!

I'm not suggesting people get sloppy in committing stuff, but I'm
suggesting to be less anal about pre-emptive reviewing and let the
testing infrastructure (or our users!) tell you what's wrong.

We currently support two OSs (with various versions of distributions,
libraries and compilers!) and three CPU architectures, but this is very
much likely to grow in the future. And permutations grow quadratically,
not linearly!

There will be a time in the future when potentially hundreds of machines
will run a head-less 'building' cruisecontrol (or equivalent) system
that will dump results in real time into a common repository (at
harmonytest.org probably) and such repository will perform analysis of
the results and find who to blame and send an email to the person with
the problem and copying the dev mail list.

A developers commit could run the tests fine on their platform and break
every other one... their own testing would have been completely useless
anyway.

-- 
Stefano.


Mime
View raw message