harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Varlamov" <alexey.v.varla...@gmail.com>
Subject Re: [drlvm][unit] 100% of class library tests pass
Date Thu, 16 Nov 2006 09:57:03 GMT
2006/11/16, Tim Ellison <t.p.ellison@gmail.com>:
> Pavel Ozhdikhin wrote:
> > We have to evolving systems - classlib and DRLVM. To check commits to
> > classlib we need a stable DRLVM which can pass 100% of HUT. Otherwise it's
> > impossible to use DRLVM for pre-commit testing - you never know whether
> > your
> > test fail because of your patch or due to latest changes in DRLVM.
> >
> > I remember the time when DRLVM and Jitrino actively evolved - for some time
> > JIT had to use an older version of DRLVM which could pass all commit
> > criteria because newer versions suffered from regressions. And finally we
> > came to comon strict commit criteria which prevented regressions in both VM
> > and JIT.
> >
> > To avoid regressions using DRLVM in classlib testing I see 3 possible
> > solutions:
> >
> > 1. Use one fixed DRLVM version which can pass 100% HUT test. Update this
> > version from time to time.
> >    Pros: + Less time to run DRLVM pre-commit tests
> >              + Classlib does not suffer from regressions in DRLVM
> >    Cons: - DRLVM will suffer from regressions
> >               - Classlib can not use the latest DRLVM
> >               - Need additional efforts to regain stability on DRLVM
> >                 when we want to update the version for classlib testing
> >
> > 2. Add HUT to CruiseControl testing on DRLVM and rollback commits causing
> > regressions
> >    Pros: + Less time to run DRLVM pre-commit tests
> >              + Classlib can use the latest DRLVM
> >    Cons: - Classlib can suffer from DRLVM regressions (time lag before
> > rollback)
> >               - It is not always clear which commit caused a regression
> >               - Rollbacks are costly
> >
> > 3. Add HUT to the commit criteria for DRLVM
> >     Pros: + Classlib always can use the latest DRLVM
> >               + DRLVM has no regressions regarding to HUT
> >     Cons: - More time to run DRLVM pre-commit tests (I was told that HUT
> >                   take 25 minutes running in single JVM mode)
> >
> > I think that preventing a problem is better than solving it afterwards. So,
> > I personally would choose the 3rd approach, don't mind against the second
> > and dislike the first one. Probably some combination of these is possible.
>
> While I appreciate the desire to keep things stable, I think it is
> unreasonable to ask developers to run the entire test suite each time.
> As we have seen in the classlib code, running targeted tests before
> commit and leaving the build machines to run continuous tests ensures
> that we are productive and are notified of breakages in time to easily
> back out a patch and re-evaluate.
>
> With the amount of machine time we have running harmony tests on
> different cpu's/os's/compilers/etc we are getting better coverage than
> any individual could be expected to provide.
>
> Which is a long way of saying I think option (2) above is best -- and
> relies on the bid machines letting us know if things break, and the
> commitment from all of us to go straight in and fix it.

I can't say it better. Thank you Tim :)
Maybe just to reinforce:
1) We have absolutely stable model VM for classlib verification - j9
it's name. Therefore I really don't think DRLVM can affect classlib's
progress disruptively.
2) Yes, there are times when some component advances in leaps as
against accustomed smooth evolution. I do believe we'll be able to
manage such cases individually, w/o overburdening everyday activities.

>
> Regards,
> Tim
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>

Mime
View raw message