harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Loenko" <mloe...@gmail.com>
Subject Re: [classlib][build] exclude list impl issues
Date Tue, 30 Jan 2007 09:06:44 GMT
2007/1/30, Geir Magnusson Jr. <geir@pobox.com>:
>
> On Jan 30, 2007, at 7:40 AM, Mikhail Loenko wrote:
>
> > 2007/1/30, Geir Magnusson Jr. <geir@pobox.com>:
> >>
> >> On Jan 30, 2007, at 5:17 AM, Mikhail Loenko wrote:
> >>
> >> > 2007/1/29, Geir Magnusson Jr. <geir@pobox.com>:
> >> >>
> >> >> On Jan 29, 2007, at 8:45 AM, Vladimir Ivanov wrote:
> >> >>
> >> >> > On 1/29/07, Geir Magnusson Jr. <geir@pobox.com> wrote:
> >> >> >>
> >> >> >>
> >> >> >> On Jan 28, 2007, at 11:28 PM, Vladimir Ivanov wrote:
> >> >> >>
> >> >> >> > OK. Current changes for exclude lists were integrated
and
> >> now we
> >> >> >> have
> >> >> >> > 3-level exclude list:
> >> >> >> >
> >> >> >> > exlude.common - tests, that failed over all platforms
> >> >> >> >
> >> >> >> > exclude.<platform> - tests, that failed over specific
> >> >> platform only
> >> >> >> >
> >> >> >> > exclude.<platform>.interm - tests that failed time
to time
> >> over
> >> >> >> > specific
> >> >> >> > platform only.
> >> >> >>
> >> >> >> Quick q - why separate interm out?  why not just put in the
> >> >> platform
> >> >> >> file?
> >> >> >
> >> >> >
> >> >> > To do exclude lists clean up process easier. To delete test from
> >> >> the
> >> >> > platform list you should check that this test passed one
> >> time. To
> >> >> > delete
> >> >> > test from 'intermittently failed' exclude list each test
> >> should be
> >> >> > run in
> >> >> > cycle.
> >> >>
> >> >> :/
> >> >>
> >> >> Doesn't the first case simply mean you got lucky?
> >> >
> >> > Well if some permanently failing tests starts to pass then it's
> >> > probably more than
> >> > just luck
> >>
> >> Of course - someone did work (although historically we know that
> >> we're not always sure why)
> >>
> >> A ha.  You're suggesting a test can't be on both lists?  That was the
> >> logical conundrum - if you had the same test on both lists, then it's
> >> clear it passes *sometimes*.
> >
> > Well, if the test passes sometimes, then let's don't have such
> > tests :)
>
> I hope that's a joke, along the lines of :
>
> Patient : It hurts when I lift my arm
> Doctor : Then don't lift your arm.
>
> > Probably it should be in the "permanent" list with a proper comment
>
> That was what I was getting at - if there is a test that is
> intermittent, then IMO it's a failure, and should be on the main list
> (with a comment, maybe, and a JIRA to track it).  Therefore, I don't
> understand what tests you'd then need to put on the "intermittent" list.
>
> See why I'm confused?

I see.

Before we started many different CC runs, we did not have problems with
intermittent failures: sometimes some tests failed but that was not a problem:
we re-run the tests and committed.

Now we know that many tests have intermittent problems.

Remember, that we exclude not just single methods but whole files? So
that means that if we have some intermittent failure in one test
method then we exclude
the whole file form the pre-commit checks.

So we have options:
1. exclude the tests from both CC and pre-commit testing and
2. exclude them from CC and keep for pre-commit

Given that it's a big bunch of tests, IMHO it makes sense to keep
them for pre-commit testing

Thanks,
Mikhail

>
> geir
>
> >
> > Thanks,
> > Mikhail
> >
> >>
> >> >
> >> > I'm for separating these x-lists: we may get rid of CC failures,
> >> and
> >> > at the same time
> >> > those tests will still find regressions (when run pre-commit tests)
> >> >
> >> > Thanks,
> >> > Mikhail
> >> >
> >> >
> >> >>
> >> >> >
> >> >> >>
> >> >> >> > Any file in this chain may be skipped. Final exclude
lists
> >> are
> >> >> >> > generated at
> >> >> >> > the build time and stored to the ${hy.hdk}/build directory.
> >> >> Thanks
> >> >> >> > to Alexei
> >> >> >> > Zakharov for this changes.
> >> >> >>
> >> >> >> Another quick q - why not just glom things together in memory?
> >> >> >
> >> >> >
> >> >> >
> >> >> > If we will run tests against HDK it will nice to skip excluded
> >> >> > tests. For
> >> >> > this reason exclude lists should be stored somewhere in the
> >> built
> >> >> > product.
> >> >> >
> >> >>
> >> >> That makes perfect sense.  Thanks
> >> >>
> >> >> geir
> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> > But I have one more question to discuss: should we use
the
> >> tests
> >> >> >> from
> >> >> >> > 'intermittently failed' exclude list for preintegration
> >> testing?
> >> >> >>
> >> >> >> What does that mean?
> >> >> >
> >> >> >
> >> >> >
> >> >> > OK. Now pre-integration test include 'ant test' with
> >> requirements
> >> >> > that all
> >> >> > tests should be passed. While we have intermittently failed
> >> >> tests this
> >> >> > target sometimes report 'failed' status for these tests without
> >> >> > correlation
> >> >> > with commit changes. We have 2 options here:
> >> >> >
> >> >> > - exclude these tests in the platform-specific ex-list
> >> >> >
> >> >> > - exclude these tests in the special ex-list
> >> >> >
> >> >> > Each option has pro and contra: in the first case we can miss
> >> the
> >> >> > regression
> >> >> > when intermittently failed test became always failed or
> >> delete test
> >> >> > from
> >> >> > exclude lists if it passed only one time.
> >> >> >
> >> >> > In the second case we have some overhead to support special
> >> exclude
> >> >> > lists.
> >> >> >
> >> >> > Note to detect regression the 'default' mode should be 'on (run)
> >> >> > intermittently failed tests' to test commit changes and 'off'
to
> >> >> > run tests
> >> >> > under CC.
> >> >> >
> >> >> > I'll add switches to on/off levels for resulting exclude lists.
> >> >> >
> >> >> > thanks, Vladimir
> >> >> >
> >> >> >
> >> >> >
> >> >> >> > If we use
> >> >> >> > it we may miss some regression when intermittently failed
> >> >> test will
> >> >> >> > failed
> >> >> >> > constantly however if we does not use it we need to run
test
> >> >> twice
> >> >> >> > sometimes.
> >> >> >> > What is correct behavior?
> >> >> >> >
> >> >> >> >  thanks, Vladimir
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On 1/28/07, Alex Blewitt <alex.blewitt@gmail.com>
wrote:
> >> >> >> >>
> >> >> >> >> Yeah, +1 for using common exclude lists. It makes
it easier
> >> >> when
> >> >> >> >> Harmony gets ported to other operating systems. And
I
> >> don't see
> >> >> >> the
> >> >> >> >> benefit of having empty lists in that case; and if
nothing's
> >> >> >> failing,
> >> >> >> >> you don't need an empty list either :-)
> >> >> >> >>
> >> >> >> >> Alex.
> >> >> >> >>
> >> >> >> >> On 28/01/07, Alexey Petrenko <alexey.a.petrenko@gmail.com>
> >> >> wrote:
> >> >> >> >> > +1 from me for using common exclude lists and
removing
> >> empty
> >> >> >> lists.
> >> >> >> >> >
> >> >> >> >> > SY, Alexey
> >> >> >> >> >
> >> >> >> >> > 2007/1/16, Alexey Varlamov <alexey.v.varlamov@gmail.com>:
> >> >> >> >> > > Folks,
> >> >> >> >> > >
> >> >> >> >> > > I've some concerns with recent updates
for x-list
> >> >> support: can
> >> >> >> >> we use
> >> >> >> >> > > a common macros instead of copy-pasting
4 new targets
> >> into
> >> >> >> >> build.xml
> >> >> >> >> > > for each module?
> >> >> >> >> > > Or, if we can neglect creating a compiled
x-file
> >> >> >> >> > > (${hy.hdk}/build/<module>.exclude),
just use "if"
> >> >> attribute of
> >> >> >> >> > > <excludesfile>, like this:
> >> >> >> >> > >
> >> >> >> >> > > <available property="x.list.exist"
> >> >> >> >> > > file="exclude.${hy.platform}.${hy.test.vm.name}"/>
> >> >> >> >> > > ...
> >> >> >> >> > >             <batchtest>
> >> >> >> >> > >                 <fileset dir="${src.test.java}">
> >> >> >> >> > >                     <include name="**/*Test.java"/>
> >> >> >> >> > >                     <excludesfile
> >> name="exclude.common"/>
> >> >> >> >> > >                     <excludesfile name="${exclude.file}
> >> >> if="
> >> >> >> >> x.list.exist" />
> >> >> >> >> > >                 </fileset>
> >> >> >> >> > >             </batchtest>
> >> >> >> >> > >
> >> >> >> >> > > Also, I suggest to delete empty x-list
remained after
> >> >> >> introducing
> >> >> >> >> common lists.
> >> >> >> >> > >
> >> >> >> >> > > Another issue is with "hy.test.vm.name",
it was a
> >> >> surprise for
> >> >> >> >> me that
> >> >> >> >> > > it is not autodetected yet. Most obvious
way to get
> >> it is
> >> >> >> to read
> >> >> >> >> > > "java.vm.name" property, this only requires
running
> >> trivial
> >> >> >> test.
> >> >> >> >> > >
> >> >> >> >> > > --
> >> >> >> >> > > Alexey
> >> >> >> >> > >
> >> >> >> >> >
> >> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
>
>

Mime
View raw message