harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Loenko" <mloe...@gmail.com>
Subject Re: [classlib][build] exclude list impl issues
Date Tue, 30 Jan 2007 07:40:39 GMT
2007/1/30, Geir Magnusson Jr. <geir@pobox.com>:
>
> On Jan 30, 2007, at 5:17 AM, Mikhail Loenko wrote:
>
> > 2007/1/29, Geir Magnusson Jr. <geir@pobox.com>:
> >>
> >> On Jan 29, 2007, at 8:45 AM, Vladimir Ivanov wrote:
> >>
> >> > On 1/29/07, Geir Magnusson Jr. <geir@pobox.com> wrote:
> >> >>
> >> >>
> >> >> On Jan 28, 2007, at 11:28 PM, Vladimir Ivanov wrote:
> >> >>
> >> >> > OK. Current changes for exclude lists were integrated and now
we
> >> >> have
> >> >> > 3-level exclude list:
> >> >> >
> >> >> > exlude.common - tests, that failed over all platforms
> >> >> >
> >> >> > exclude.<platform> - tests, that failed over specific
> >> platform only
> >> >> >
> >> >> > exclude.<platform>.interm - tests that failed time to time
over
> >> >> > specific
> >> >> > platform only.
> >> >>
> >> >> Quick q - why separate interm out?  why not just put in the
> >> platform
> >> >> file?
> >> >
> >> >
> >> > To do exclude lists clean up process easier. To delete test from
> >> the
> >> > platform list you should check that this test passed one time. To
> >> > delete
> >> > test from 'intermittently failed' exclude list each test should be
> >> > run in
> >> > cycle.
> >>
> >> :/
> >>
> >> Doesn't the first case simply mean you got lucky?
> >
> > Well if some permanently failing tests starts to pass then it's
> > probably more than
> > just luck
>
> Of course - someone did work (although historically we know that
> we're not always sure why)
>
> A ha.  You're suggesting a test can't be on both lists?  That was the
> logical conundrum - if you had the same test on both lists, then it's
> clear it passes *sometimes*.

Well, if the test passes sometimes, then let's don't have such tests :)
Probably it should be in the "permanent" list with a proper comment

Thanks,
Mikhail

>
> >
> > I'm for separating these x-lists: we may get rid of CC failures, and
> > at the same time
> > those tests will still find regressions (when run pre-commit tests)
> >
> > Thanks,
> > Mikhail
> >
> >
> >>
> >> >
> >> >>
> >> >> > Any file in this chain may be skipped. Final exclude lists are
> >> >> > generated at
> >> >> > the build time and stored to the ${hy.hdk}/build directory.
> >> Thanks
> >> >> > to Alexei
> >> >> > Zakharov for this changes.
> >> >>
> >> >> Another quick q - why not just glom things together in memory?
> >> >
> >> >
> >> >
> >> > If we will run tests against HDK it will nice to skip excluded
> >> > tests. For
> >> > this reason exclude lists should be stored somewhere in the built
> >> > product.
> >> >
> >>
> >> That makes perfect sense.  Thanks
> >>
> >> geir
> >>
> >> >>
> >> >> >
> >> >> >
> >> >> > But I have one more question to discuss: should we use the tests
> >> >> from
> >> >> > 'intermittently failed' exclude list for preintegration testing?
> >> >>
> >> >> What does that mean?
> >> >
> >> >
> >> >
> >> > OK. Now pre-integration test include 'ant test' with requirements
> >> > that all
> >> > tests should be passed. While we have intermittently failed
> >> tests this
> >> > target sometimes report 'failed' status for these tests without
> >> > correlation
> >> > with commit changes. We have 2 options here:
> >> >
> >> > - exclude these tests in the platform-specific ex-list
> >> >
> >> > - exclude these tests in the special ex-list
> >> >
> >> > Each option has pro and contra: in the first case we can miss the
> >> > regression
> >> > when intermittently failed test became always failed or delete test
> >> > from
> >> > exclude lists if it passed only one time.
> >> >
> >> > In the second case we have some overhead to support special exclude
> >> > lists.
> >> >
> >> > Note to detect regression the 'default' mode should be 'on (run)
> >> > intermittently failed tests' to test commit changes and 'off' to
> >> > run tests
> >> > under CC.
> >> >
> >> > I'll add switches to on/off levels for resulting exclude lists.
> >> >
> >> > thanks, Vladimir
> >> >
> >> >
> >> >
> >> >> > If we use
> >> >> > it we may miss some regression when intermittently failed
> >> test will
> >> >> > failed
> >> >> > constantly however if we does not use it we need to run test
> >> twice
> >> >> > sometimes.
> >> >> > What is correct behavior?
> >> >> >
> >> >> >  thanks, Vladimir
> >> >> >
> >> >> >
> >> >> >
> >> >> > On 1/28/07, Alex Blewitt <alex.blewitt@gmail.com> wrote:
> >> >> >>
> >> >> >> Yeah, +1 for using common exclude lists. It makes it easier
> >> when
> >> >> >> Harmony gets ported to other operating systems. And I don't
see
> >> >> the
> >> >> >> benefit of having empty lists in that case; and if nothing's
> >> >> failing,
> >> >> >> you don't need an empty list either :-)
> >> >> >>
> >> >> >> Alex.
> >> >> >>
> >> >> >> On 28/01/07, Alexey Petrenko <alexey.a.petrenko@gmail.com>
> >> wrote:
> >> >> >> > +1 from me for using common exclude lists and removing
empty
> >> >> lists.
> >> >> >> >
> >> >> >> > SY, Alexey
> >> >> >> >
> >> >> >> > 2007/1/16, Alexey Varlamov <alexey.v.varlamov@gmail.com>:
> >> >> >> > > Folks,
> >> >> >> > >
> >> >> >> > > I've some concerns with recent updates for x-list
> >> support: can
> >> >> >> we use
> >> >> >> > > a common macros instead of copy-pasting 4 new targets
into
> >> >> >> build.xml
> >> >> >> > > for each module?
> >> >> >> > > Or, if we can neglect creating a compiled x-file
> >> >> >> > > (${hy.hdk}/build/<module>.exclude), just use
"if"
> >> attribute of
> >> >> >> > > <excludesfile>, like this:
> >> >> >> > >
> >> >> >> > > <available property="x.list.exist"
> >> >> >> > > file="exclude.${hy.platform}.${hy.test.vm.name}"/>
> >> >> >> > > ...
> >> >> >> > >             <batchtest>
> >> >> >> > >                 <fileset dir="${src.test.java}">
> >> >> >> > >                     <include name="**/*Test.java"/>
> >> >> >> > >                     <excludesfile name="exclude.common"/>
> >> >> >> > >                     <excludesfile name="${exclude.file}
> >> if="
> >> >> >> x.list.exist" />
> >> >> >> > >                 </fileset>
> >> >> >> > >             </batchtest>
> >> >> >> > >
> >> >> >> > > Also, I suggest to delete empty x-list remained
after
> >> >> introducing
> >> >> >> common lists.
> >> >> >> > >
> >> >> >> > > Another issue is with "hy.test.vm.name", it was
a
> >> surprise for
> >> >> >> me that
> >> >> >> > > it is not autodetected yet. Most obvious way to
get it is
> >> >> to read
> >> >> >> > > "java.vm.name" property, this only requires running
trivial
> >> >> test.
> >> >> >> > >
> >> >> >> > > --
> >> >> >> > > Alexey
> >> >> >> > >
> >> >> >> >
> >> >> >>
> >> >>
> >> >>
> >>
> >>
>
>

Mime
View raw message