harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Loenko" <mloe...@gmail.com>
Subject Re: [drlvm][unit] 100% of class library tests pass
Date Thu, 16 Nov 2006 11:42:50 GMT
16 Nov 2006 17:15:14 +0600, Egor Pasko <egor.pasko@gmail.com>:
> On the 0x223 day of Apache Harmony Alexey Varlamov wrote:
> > 2006/11/16, Tim Ellison <t.p.ellison@gmail.com>:
> > > Pavel Ozhdikhin wrote:
> > > > We have to evolving systems - classlib and DRLVM. To check commits to
> > > > classlib we need a stable DRLVM which can pass 100% of HUT. Otherwise
it's
> > > > impossible to use DRLVM for pre-commit testing - you never know whether
> > > > your
> > > > test fail because of your patch or due to latest changes in DRLVM.
> > > >
> > > > I remember the time when DRLVM and Jitrino actively evolved - for some
time
> > > > JIT had to use an older version of DRLVM which could pass all commit
> > > > criteria because newer versions suffered from regressions. And finally
we
> > > > came to comon strict commit criteria which prevented regressions in both
VM
> > > > and JIT.
> > > >
> > > > To avoid regressions using DRLVM in classlib testing I see 3 possible
> > > > solutions:
> > > >
> > > > 1. Use one fixed DRLVM version which can pass 100% HUT test. Update this
> > > > version from time to time.
> > > >    Pros: + Less time to run DRLVM pre-commit tests
> > > >              + Classlib does not suffer from regressions in DRLVM
> > > >    Cons: - DRLVM will suffer from regressions
> > > >               - Classlib can not use the latest DRLVM
> > > >               - Need additional efforts to regain stability on DRLVM
> > > >                 when we want to update the version for classlib testing
> > > >
> > > > 2. Add HUT to CruiseControl testing on DRLVM and rollback commits causing
> > > > regressions
> > > >    Pros: + Less time to run DRLVM pre-commit tests
> > > >              + Classlib can use the latest DRLVM
> > > >    Cons: - Classlib can suffer from DRLVM regressions (time lag before
> > > > rollback)
> > > >               - It is not always clear which commit caused a regression
> > > >               - Rollbacks are costly
> > > >
> > > > 3. Add HUT to the commit criteria for DRLVM
> > > >     Pros: + Classlib always can use the latest DRLVM
> > > >               + DRLVM has no regressions regarding to HUT
> > > >     Cons: - More time to run DRLVM pre-commit tests (I was told that HUT
> > > >                   take 25 minutes running in single JVM mode)
> > > >
> > > > I think that preventing a problem is better than solving it afterwards.
So,
> > > > I personally would choose the 3rd approach, don't mind against the second
> > > > and dislike the first one. Probably some combination of these is possible.
> > >
> > > While I appreciate the desire to keep things stable, I think it is
> > > unreasonable to ask developers to run the entire test suite each time.
> > > As we have seen in the classlib code, running targeted tests before
> > > commit and leaving the build machines to run continuous tests ensures
> > > that we are productive and are notified of breakages in time to easily
> > > back out a patch and re-evaluate.
> > >
> > > With the amount of machine time we have running harmony tests on
> > > different cpu's/os's/compilers/etc we are getting better coverage than
> > > any individual could be expected to provide.
> >
> > > Which is a long way of saying I think option (2) above is best -- and
> > > relies on the bid machines letting us know if things break, and the
> > > commitment from all of us to go straight in and fix it.
> >
> > I can't say it better. Thank you Tim :)
> > Maybe just to reinforce:
> > 1) We have absolutely stable model VM for classlib verification - j9
> > it's name. Therefore I really don't think DRLVM can affect classlib's
> > progress disruptively.
> > 2) Yes, there are times when some component advances in leaps as
> > against accustomed smooth evolution. I do believe we'll be able to
> > manage such cases individually, w/o overburdening everyday activities.
>
> I am for (2) too.

+1

And I hope we will have other workloads running on Harmony nightly and
reporting
regressions to the list.


But a small correction: rollback is not always
> reasonable. We can explicitly agree if we do rollback or fix ASAP (as
> we did with TM and launcher).
+1


Thanks,
Mikhail



>
> I do not like (1) because interfaces are evolving. And I am not
> feeling like this process would stop in a short time. (3) is just too
> long to run as pre-commit, IMHO.
>
> Though, we might select some most bug-catching tests from HUT to run
> as pre-commit. I have nothing concrete to suggest now. We might return
> to this idea in future, when we have a longer history.
>
> --
> Egor Pasko
>
>

Mime
View raw message