harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Egor Pasko <egor.pa...@gmail.com>
Subject Re: [drlvm][unit] 100% of class library tests pass
Date Thu, 16 Nov 2006 11:15:14 GMT
On the 0x223 day of Apache Harmony Alexey Varlamov wrote:
> 2006/11/16, Tim Ellison <t.p.ellison@gmail.com>:
> > Pavel Ozhdikhin wrote:
> > > We have to evolving systems - classlib and DRLVM. To check commits to
> > > classlib we need a stable DRLVM which can pass 100% of HUT. Otherwise it's
> > > impossible to use DRLVM for pre-commit testing - you never know whether
> > > your
> > > test fail because of your patch or due to latest changes in DRLVM.
> > >
> > > I remember the time when DRLVM and Jitrino actively evolved - for some time
> > > JIT had to use an older version of DRLVM which could pass all commit
> > > criteria because newer versions suffered from regressions. And finally we
> > > came to comon strict commit criteria which prevented regressions in both VM
> > > and JIT.
> > >
> > > To avoid regressions using DRLVM in classlib testing I see 3 possible
> > > solutions:
> > >
> > > 1. Use one fixed DRLVM version which can pass 100% HUT test. Update this
> > > version from time to time.
> > >    Pros: + Less time to run DRLVM pre-commit tests
> > >              + Classlib does not suffer from regressions in DRLVM
> > >    Cons: - DRLVM will suffer from regressions
> > >               - Classlib can not use the latest DRLVM
> > >               - Need additional efforts to regain stability on DRLVM
> > >                 when we want to update the version for classlib testing
> > >
> > > 2. Add HUT to CruiseControl testing on DRLVM and rollback commits causing
> > > regressions
> > >    Pros: + Less time to run DRLVM pre-commit tests
> > >              + Classlib can use the latest DRLVM
> > >    Cons: - Classlib can suffer from DRLVM regressions (time lag before
> > > rollback)
> > >               - It is not always clear which commit caused a regression
> > >               - Rollbacks are costly
> > >
> > > 3. Add HUT to the commit criteria for DRLVM
> > >     Pros: + Classlib always can use the latest DRLVM
> > >               + DRLVM has no regressions regarding to HUT
> > >     Cons: - More time to run DRLVM pre-commit tests (I was told that HUT
> > >                   take 25 minutes running in single JVM mode)
> > >
> > > I think that preventing a problem is better than solving it afterwards. So,
> > > I personally would choose the 3rd approach, don't mind against the second
> > > and dislike the first one. Probably some combination of these is possible.
> >
> > While I appreciate the desire to keep things stable, I think it is
> > unreasonable to ask developers to run the entire test suite each time.
> > As we have seen in the classlib code, running targeted tests before
> > commit and leaving the build machines to run continuous tests ensures
> > that we are productive and are notified of breakages in time to easily
> > back out a patch and re-evaluate.
> >
> > With the amount of machine time we have running harmony tests on
> > different cpu's/os's/compilers/etc we are getting better coverage than
> > any individual could be expected to provide.
>
> > Which is a long way of saying I think option (2) above is best -- and
> > relies on the bid machines letting us know if things break, and the
> > commitment from all of us to go straight in and fix it.
> 
> I can't say it better. Thank you Tim :)
> Maybe just to reinforce:
> 1) We have absolutely stable model VM for classlib verification - j9
> it's name. Therefore I really don't think DRLVM can affect classlib's
> progress disruptively.
> 2) Yes, there are times when some component advances in leaps as
> against accustomed smooth evolution. I do believe we'll be able to
> manage such cases individually, w/o overburdening everyday activities.

I am for (2) too. But a small correction: rollback is not always
reasonable. We can explicitly agree if we do rollback or fix ASAP (as
we did with TM and launcher).

I do not like (1) because interfaces are evolving. And I am not
feeling like this process would stop in a short time. (3) is just too
long to run as pre-commit, IMHO.

Though, we might select some most bug-catching tests from HUT to run
as pre-commit. I have nothing concrete to suggest now. We might return
to this idea in future, when we have a longer history.

-- 
Egor Pasko


Mime
View raw message