harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fedotov, Alexei A" <alexei.a.fedo...@intel.com>
Subject RE: [drlvm][unit] 100% of class library tests pass
Date Thu, 16 Nov 2006 10:08:10 GMT
Pavel, All,

Let me add one "pro" for the second approach: it works already.
Vladimir's scripts daily upload test results to http://harmonytest.org.

With best regards,
Alexei Fedotov,
Intel Java & XML Engineering

>-----Original Message-----
>From: Tim Ellison [mailto:t.p.ellison@gmail.com]
>Sent: Thursday, November 16, 2006 12:37 PM
>To: harmony-dev@incubator.apache.org
>Subject: Re: [drlvm][unit] 100% of class library tests pass
>
>Pavel Ozhdikhin wrote:
>> We have to evolving systems - classlib and DRLVM. To check commits to
>> classlib we need a stable DRLVM which can pass 100% of HUT. Otherwise
>it's
>> impossible to use DRLVM for pre-commit testing - you never know
whether
>> your
>> test fail because of your patch or due to latest changes in DRLVM.
>>
>> I remember the time when DRLVM and Jitrino actively evolved - for
some
>time
>> JIT had to use an older version of DRLVM which could pass all commit
>> criteria because newer versions suffered from regressions. And
finally we
>> came to comon strict commit criteria which prevented regressions in
both
>VM
>> and JIT.
>>
>> To avoid regressions using DRLVM in classlib testing I see 3 possible
>> solutions:
>>
>> 1. Use one fixed DRLVM version which can pass 100% HUT test. Update
this
>> version from time to time.
>>    Pros: + Less time to run DRLVM pre-commit tests
>>              + Classlib does not suffer from regressions in DRLVM
>>    Cons: - DRLVM will suffer from regressions
>>               - Classlib can not use the latest DRLVM
>>               - Need additional efforts to regain stability on DRLVM
>>                 when we want to update the version for classlib
testing
>>
>> 2. Add HUT to CruiseControl testing on DRLVM and rollback commits
causing
>> regressions
>>    Pros: + Less time to run DRLVM pre-commit tests
>>              + Classlib can use the latest DRLVM
>>    Cons: - Classlib can suffer from DRLVM regressions (time lag
before
>> rollback)
>>               - It is not always clear which commit caused a
regression
>>               - Rollbacks are costly
>>
>> 3. Add HUT to the commit criteria for DRLVM
>>     Pros: + Classlib always can use the latest DRLVM
>>               + DRLVM has no regressions regarding to HUT
>>     Cons: - More time to run DRLVM pre-commit tests (I was told that
HUT
>>                   take 25 minutes running in single JVM mode)
>>
>> I think that preventing a problem is better than solving it
afterwards.
>So,
>> I personally would choose the 3rd approach, don't mind against the
second
>> and dislike the first one. Probably some combination of these is
>possible.
>
>While I appreciate the desire to keep things stable, I think it is
>unreasonable to ask developers to run the entire test suite each time.
>As we have seen in the classlib code, running targeted tests before
>commit and leaving the build machines to run continuous tests ensures
>that we are productive and are notified of breakages in time to easily
>back out a patch and re-evaluate.
>
>With the amount of machine time we have running harmony tests on
>different cpu's/os's/compilers/etc we are getting better coverage than
>any individual could be expected to provide.
>
>Which is a long way of saying I think option (2) above is best -- and
>relies on the bid machines letting us know if things break, and the
>commitment from all of us to go straight in and fix it.
>
>Regards,
>Tim
>
>--
>
>Tim Ellison (t.p.ellison@gmail.com)
>IBM Java technology centre, UK.

Mime
View raw message