harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Ozhdikhin" <pavel.ozhdik...@gmail.com>
Subject Re: [drlvm][unit] 100% of class library tests pass
Date Thu, 16 Nov 2006 08:24:50 GMT
We have to evolving systems - classlib and DRLVM. To check commits to
classlib we need a stable DRLVM which can pass 100% of HUT. Otherwise it's
impossible to use DRLVM for pre-commit testing - you never know whether your
test fail because of your patch or due to latest changes in DRLVM.

I remember the time when DRLVM and Jitrino actively evolved - for some time
JIT had to use an older version of DRLVM which could pass all commit
criteria because newer versions suffered from regressions. And finally we
came to comon strict commit criteria which prevented regressions in both VM
and JIT.

To avoid regressions using DRLVM in classlib testing I see 3 possible
solutions:

1. Use one fixed DRLVM version which can pass 100% HUT test. Update this
version from time to time.
    Pros: + Less time to run DRLVM pre-commit tests
              + Classlib does not suffer from regressions in DRLVM
    Cons: - DRLVM will suffer from regressions
               - Classlib can not use the latest DRLVM
               - Need additional efforts to regain stability on DRLVM
                 when we want to update the version for classlib testing

2. Add HUT to CruiseControl testing on DRLVM and rollback commits causing
regressions
    Pros: + Less time to run DRLVM pre-commit tests
              + Classlib can use the latest DRLVM
    Cons: - Classlib can suffer from DRLVM regressions (time lag before
rollback)
               - It is not always clear which commit caused a regression
               - Rollbacks are costly

3. Add HUT to the commit criteria for DRLVM
     Pros: + Classlib always can use the latest DRLVM
               + DRLVM has no regressions regarding to HUT
     Cons: - More time to run DRLVM pre-commit tests (I was told that HUT
                   take 25 minutes running in single JVM mode)

I think that preventing a problem is better than solving it afterwards. So,
I personally would choose the 3rd approach, don't mind against the second
and dislike the first one. Probably some combination of these is possible.

thanks,
Pavel

On 11/16/06, Alexey Varlamov <alexey.v.varlamov@gmail.com> wrote:
>
> 2006/11/16, Mikhail Loenko <mloenko@gmail.com>:
> > why not?
> But what benefit it would bring? build test in DRLVM takes too much
> time already, I'm afraid people will just stop using it :(
>
> This is analogous to enforcing full testing in classlib for every
> change regardless of module. Evidently this is too expensive.
>
> >
> > 2006/11/16, Geir Magnusson Jr. <geir@pobox.com>:
> > >
> > >
> > > Pavel Ozhdikhin wrote:
> > > > On 11/16/06, Geir Magnusson Jr. <geir@pobox.com> wrote:
> > > >>
> > > >> Be sure to not miss anyone :)  This was a great community effort,
> with
> > > >> everyone pitching in.
> > > >>
> > > >> DRLVM is now a full peer  to J9 in Harmony testing.  :)  We still
> need
> > > >> to use J9 (and another VM that happens to work with our
> classlibrary),
> > > >> as a sanity check, but we should from now on use DRLVM in our CI
> testing
> > > >> framework.
> > > >
> > > >
> > > > On the other hand, to make sure DRLVM has no regressions regarding
> > > > to Classlibrary Unit Tests it makes sense to add these tests to the
> "test"
> > > > target of DRLVM build.
> > > > What do people think?
> > >
> > > Adding classlib unit tests to DRLVM test target?  No thanks :)
> > >
> > > geir
> > >
> > > >
> > > > Thanks,
> > > > Pavel
> > > >
> > > >
> > > >
> > > >> geir
> > > >>
> > > >>
> > > >> Alexei Fedotov wrote:
> > > >> > Oops, I've missed:
> > > >> > * Andrew Zhang for reviewing class library patches and helpful
> > > >> discussions
> > > >> >
> > > >> > On 11/16/06, Alexei Fedotov <alexei.fedotov@gmail.com>
wrote:
> > > >> >>
> > > >> >> Folks,
> > > >> >> According to http://harmonytest.org, today 100% of class
library
> unit
> > > >> >> tests pass on DRLVM. Thank you all! It takes 44 days for
the
> great
> > > >> >> team we are. Thanks for your thoughtful, diligent work and
deep
> > > >> >> inspiration. Kudos to you for the following (and not limited
to
> this):
> > > >> >>
> > > >> >> * Alexey Varlamov and Elena for driving the whole process
> > > >> >> * Anton and Vladimir Ivanov for automating test runs
> > > >> >> * Geir and Gregory for checking and committing related DRLVM
> patches
> > > >> >> * Paulex, Tim, Nathan, Stepan and Mikhail Loenko for checking
> and
> > > >> >> committing related class library patches
> > > >> >> * Alexey Petrenko for becoming ICU expert and writing good
JIRA
> issue
> > > >> >> resolution guidelines
> > > >> >> * Alexei Zakharov for resolving class library test issues
> > > >> >> * Ilya Okomin, Denis Kishenko, Oleg Khaschansky and Alexey
> Ivanov for
> > > >> >> fixing class library and tests* Ivan, Egor, Mikhail Fursov,
> Nikolay
> > > >> >> Sidelnikov and Alexander Astapchuk for making execution engines
> work
> > > >> >> * Tatiana and Maxim for filing JIRA issues about test failures
> > > >> >> * Nikolay Kuznetsov for completing thread interruption handling
> and
> > > >> >> reverting consequences of park/unpark integration
> > > >> >> * Pavel Afremov for fixing exception handling
> > > >> >> * Boris Kuznetsov for intelligent fixing of security tests
> > > >> >> * Rana and Salikh for evaluating and discussing problems,
> reviewing
> > > >> >> and trying DRLVM patches
> > > >> >> * Pavel Pervov and Evgueni for help with DRLVM patches
> > > >> >> * Artem for discovering and fixing weird Windows* behavior
> > > >> >> * My wife for bringing hot tea to the computer during sleepless
> nights
> > > >> >>
> > > >> >> There are still open issues with reliability, multiprocessor
and
> other
> > > >> >> special configurations, so the page
> > > >> >> http://wiki.apache.org/harmony/Unit_Tests_Pass_on_DRLVM 
remains
> > > >> >> active. But this shouldn't prevent us from including class
> library
> > > >> >> testing into Harmony "zero regression" policy. What do you
> think?
> > > >> >>
> > > >> >> --
> > > >> >> Thank you,
> > > >> >> Alexei
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message