2007/1/30, Geir Magnusson Jr. <geir@pobox.com>:
>
> On Jan 30, 2007, at 7:40 AM, Mikhail Loenko wrote:
>
> > 2007/1/30, Geir Magnusson Jr. <geir@pobox.com>:
> >>
> >> On Jan 30, 2007, at 5:17 AM, Mikhail Loenko wrote:
> >>
> >> > 2007/1/29, Geir Magnusson Jr. <geir@pobox.com>:
> >> >>
> >> >> On Jan 29, 2007, at 8:45 AM, Vladimir Ivanov wrote:
> >> >>
> >> >> > On 1/29/07, Geir Magnusson Jr. <geir@pobox.com> wrote:
> >> >> >>
> >> >> >>
> >> >> >> On Jan 28, 2007, at 11:28 PM, Vladimir Ivanov wrote:
> >> >> >>
> >> >> >> > OK. Current changes for exclude lists were integrated
and
> >> now we
> >> >> >> have
> >> >> >> > 3level exclude list:
> >> >> >> >
> >> >> >> > exlude.common  tests, that failed over all platforms
> >> >> >> >
> >> >> >> > exclude.<platform>  tests, that failed over specific
> >> >> platform only
> >> >> >> >
> >> >> >> > exclude.<platform>.interm  tests that failed time
to time
> >> over
> >> >> >> > specific
> >> >> >> > platform only.
> >> >> >>
> >> >> >> Quick q  why separate interm out? why not just put in the
> >> >> platform
> >> >> >> file?
> >> >> >
> >> >> >
> >> >> > To do exclude lists clean up process easier. To delete test from
> >> >> the
> >> >> > platform list you should check that this test passed one
> >> time. To
> >> >> > delete
> >> >> > test from 'intermittently failed' exclude list each test
> >> should be
> >> >> > run in
> >> >> > cycle.
> >> >>
> >> >> :/
> >> >>
> >> >> Doesn't the first case simply mean you got lucky?
> >> >
> >> > Well if some permanently failing tests starts to pass then it's
> >> > probably more than
> >> > just luck
> >>
> >> Of course  someone did work (although historically we know that
> >> we're not always sure why)
> >>
> >> A ha. You're suggesting a test can't be on both lists? That was the
> >> logical conundrum  if you had the same test on both lists, then it's
> >> clear it passes *sometimes*.
> >
> > Well, if the test passes sometimes, then let's don't have such
> > tests :)
>
> I hope that's a joke, along the lines of :
>
> Patient : It hurts when I lift my arm
> Doctor : Then don't lift your arm.
>
> > Probably it should be in the "permanent" list with a proper comment
>
> That was what I was getting at  if there is a test that is
> intermittent, then IMO it's a failure, and should be on the main list
> (with a comment, maybe, and a JIRA to track it). Therefore, I don't
> understand what tests you'd then need to put on the "intermittent" list.
>
> See why I'm confused?
I see.
Before we started many different CC runs, we did not have problems with
intermittent failures: sometimes some tests failed but that was not a problem:
we rerun the tests and committed.
Now we know that many tests have intermittent problems.
Remember, that we exclude not just single methods but whole files? So
that means that if we have some intermittent failure in one test
method then we exclude
the whole file form the precommit checks.
So we have options:
1. exclude the tests from both CC and precommit testing and
2. exclude them from CC and keep for precommit
Given that it's a big bunch of tests, IMHO it makes sense to keep
them for precommit testing
Thanks,
Mikhail
>
> geir
>
> >
> > Thanks,
> > Mikhail
> >
> >>
> >> >
> >> > I'm for separating these xlists: we may get rid of CC failures,
> >> and
> >> > at the same time
> >> > those tests will still find regressions (when run precommit tests)
> >> >
> >> > Thanks,
> >> > Mikhail
> >> >
> >> >
> >> >>
> >> >> >
> >> >> >>
> >> >> >> > Any file in this chain may be skipped. Final exclude
lists
> >> are
> >> >> >> > generated at
> >> >> >> > the build time and stored to the ${hy.hdk}/build directory.
> >> >> Thanks
> >> >> >> > to Alexei
> >> >> >> > Zakharov for this changes.
> >> >> >>
> >> >> >> Another quick q  why not just glom things together in memory?
> >> >> >
> >> >> >
> >> >> >
> >> >> > If we will run tests against HDK it will nice to skip excluded
> >> >> > tests. For
> >> >> > this reason exclude lists should be stored somewhere in the
> >> built
> >> >> > product.
> >> >> >
> >> >>
> >> >> That makes perfect sense. Thanks
> >> >>
> >> >> geir
> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> > But I have one more question to discuss: should we use
the
> >> tests
> >> >> >> from
> >> >> >> > 'intermittently failed' exclude list for preintegration
> >> testing?
> >> >> >>
> >> >> >> What does that mean?
> >> >> >
> >> >> >
> >> >> >
> >> >> > OK. Now preintegration test include 'ant test' with
> >> requirements
> >> >> > that all
> >> >> > tests should be passed. While we have intermittently failed
> >> >> tests this
> >> >> > target sometimes report 'failed' status for these tests without
> >> >> > correlation
> >> >> > with commit changes. We have 2 options here:
> >> >> >
> >> >> >  exclude these tests in the platformspecific exlist
> >> >> >
> >> >> >  exclude these tests in the special exlist
> >> >> >
> >> >> > Each option has pro and contra: in the first case we can miss
> >> the
> >> >> > regression
> >> >> > when intermittently failed test became always failed or
> >> delete test
> >> >> > from
> >> >> > exclude lists if it passed only one time.
> >> >> >
> >> >> > In the second case we have some overhead to support special
> >> exclude
> >> >> > lists.
> >> >> >
> >> >> > Note to detect regression the 'default' mode should be 'on (run)
> >> >> > intermittently failed tests' to test commit changes and 'off'
to
> >> >> > run tests
> >> >> > under CC.
> >> >> >
> >> >> > I'll add switches to on/off levels for resulting exclude lists.
> >> >> >
> >> >> > thanks, Vladimir
> >> >> >
> >> >> >
> >> >> >
> >> >> >> > If we use
> >> >> >> > it we may miss some regression when intermittently failed
> >> >> test will
> >> >> >> > failed
> >> >> >> > constantly however if we does not use it we need to run
test
> >> >> twice
> >> >> >> > sometimes.
> >> >> >> > What is correct behavior?
> >> >> >> >
> >> >> >> > thanks, Vladimir
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On 1/28/07, Alex Blewitt <alex.blewitt@gmail.com>
wrote:
> >> >> >> >>
> >> >> >> >> Yeah, +1 for using common exclude lists. It makes
it easier
> >> >> when
> >> >> >> >> Harmony gets ported to other operating systems. And
I
> >> don't see
> >> >> >> the
> >> >> >> >> benefit of having empty lists in that case; and if
nothing's
> >> >> >> failing,
> >> >> >> >> you don't need an empty list either :)
> >> >> >> >>
> >> >> >> >> Alex.
> >> >> >> >>
> >> >> >> >> On 28/01/07, Alexey Petrenko <alexey.a.petrenko@gmail.com>
> >> >> wrote:
> >> >> >> >> > +1 from me for using common exclude lists and
removing
> >> empty
> >> >> >> lists.
> >> >> >> >> >
> >> >> >> >> > SY, Alexey
> >> >> >> >> >
> >> >> >> >> > 2007/1/16, Alexey Varlamov <alexey.v.varlamov@gmail.com>:
> >> >> >> >> > > Folks,
> >> >> >> >> > >
> >> >> >> >> > > I've some concerns with recent updates
for xlist
> >> >> support: can
> >> >> >> >> we use
> >> >> >> >> > > a common macros instead of copypasting
4 new targets
> >> into
> >> >> >> >> build.xml
> >> >> >> >> > > for each module?
> >> >> >> >> > > Or, if we can neglect creating a compiled
xfile
> >> >> >> >> > > (${hy.hdk}/build/<module>.exclude),
just use "if"
> >> >> attribute of
> >> >> >> >> > > <excludesfile>, like this:
> >> >> >> >> > >
> >> >> >> >> > > <available property="x.list.exist"
> >> >> >> >> > > file="exclude.${hy.platform}.${hy.test.vm.name}"/>
> >> >> >> >> > > ...
> >> >> >> >> > > <batchtest>
> >> >> >> >> > > <fileset dir="${src.test.java}">
> >> >> >> >> > > <include name="**/*Test.java"/>
> >> >> >> >> > > <excludesfile
> >> name="exclude.common"/>
> >> >> >> >> > > <excludesfile name="${exclude.file}
> >> >> if="
> >> >> >> >> x.list.exist" />
> >> >> >> >> > > </fileset>
> >> >> >> >> > > </batchtest>
> >> >> >> >> > >
> >> >> >> >> > > Also, I suggest to delete empty xlist
remained after
> >> >> >> introducing
> >> >> >> >> common lists.
> >> >> >> >> > >
> >> >> >> >> > > Another issue is with "hy.test.vm.name",
it was a
> >> >> surprise for
> >> >> >> >> me that
> >> >> >> >> > > it is not autodetected yet. Most obvious
way to get
> >> it is
> >> >> >> to read
> >> >> >> >> > > "java.vm.name" property, this only requires
running
> >> trivial
> >> >> >> test.
> >> >> >> >> > >
> >> >> >> >> > > 
> >> >> >> >> > > Alexey
> >> >> >> >> > >
> >> >> >> >> >
> >> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
>
>
