harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [classlib][build] exclude list impl issues
Date Tue, 30 Jan 2007 18:04:56 GMT

On Jan 30, 2007, at 9:06 AM, Mikhail Loenko wrote:

> 2007/1/30, Geir Magnusson Jr. <geir@pobox.com>:
>>
>> On Jan 30, 2007, at 7:40 AM, Mikhail Loenko wrote:
>>
>> > 2007/1/30, Geir Magnusson Jr. <geir@pobox.com>:
>> >>
>> >> On Jan 30, 2007, at 5:17 AM, Mikhail Loenko wrote:
>> >>
>> >> > 2007/1/29, Geir Magnusson Jr. <geir@pobox.com>:
>> >> >>
>> >> >> On Jan 29, 2007, at 8:45 AM, Vladimir Ivanov wrote:
>> >> >>
>> >> >> > On 1/29/07, Geir Magnusson Jr. <geir@pobox.com> wrote:
>> >> >> >>
>> >> >> >>
>> >> >> >> On Jan 28, 2007, at 11:28 PM, Vladimir Ivanov wrote:
>> >> >> >>
>> >> >> >> > OK. Current changes for exclude lists were integrated
and
>> >> now we
>> >> >> >> have
>> >> >> >> > 3-level exclude list:
>> >> >> >> >
>> >> >> >> > exlude.common - tests, that failed over all platforms
>> >> >> >> >
>> >> >> >> > exclude.<platform> - tests, that failed over
specific
>> >> >> platform only
>> >> >> >> >
>> >> >> >> > exclude.<platform>.interm - tests that failed
time to time
>> >> over
>> >> >> >> > specific
>> >> >> >> > platform only.
>> >> >> >>
>> >> >> >> Quick q - why separate interm out?  why not just put in
the
>> >> >> platform
>> >> >> >> file?
>> >> >> >
>> >> >> >
>> >> >> > To do exclude lists clean up process easier. To delete  
>> test from
>> >> >> the
>> >> >> > platform list you should check that this test passed one
>> >> time. To
>> >> >> > delete
>> >> >> > test from 'intermittently failed' exclude list each test
>> >> should be
>> >> >> > run in
>> >> >> > cycle.
>> >> >>
>> >> >> :/
>> >> >>
>> >> >> Doesn't the first case simply mean you got lucky?
>> >> >
>> >> > Well if some permanently failing tests starts to pass then it's
>> >> > probably more than
>> >> > just luck
>> >>
>> >> Of course - someone did work (although historically we know that
>> >> we're not always sure why)
>> >>
>> >> A ha.  You're suggesting a test can't be on both lists?  That  
>> was the
>> >> logical conundrum - if you had the same test on both lists,  
>> then it's
>> >> clear it passes *sometimes*.
>> >
>> > Well, if the test passes sometimes, then let's don't have such
>> > tests :)
>>
>> I hope that's a joke, along the lines of :
>>
>> Patient : It hurts when I lift my arm
>> Doctor : Then don't lift your arm.
>>
>> > Probably it should be in the "permanent" list with a proper comment
>>
>> That was what I was getting at - if there is a test that is
>> intermittent, then IMO it's a failure, and should be on the main list
>> (with a comment, maybe, and a JIRA to track it).  Therefore, I don't
>> understand what tests you'd then need to put on the "intermittent"  
>> list.
>>
>> See why I'm confused?
>
> I see.
>
> Before we started many different CC runs, we did not have problems  
> with
> intermittent failures: sometimes some tests failed but that was not  
> a problem:
> we re-run the tests and committed.
>
> Now we know that many tests have intermittent problems.
>
> Remember, that we exclude not just single methods but whole files? So
> that means that if we have some intermittent failure in one test
> method then we exclude
> the whole file form the pre-commit checks.

I wonder if we can bang JUnit into letting us exclude single tests.

>
> So we have options:
> 1. exclude the tests from both CC and pre-commit testing and
> 2. exclude them from CC and keep for pre-commit
>
> Given that it's a big bunch of tests, IMHO it makes sense to keep
> them for pre-commit testing

I agree because as you point out, we are getting rid of a lot of good  
tests.  (Again, I wonder if we can just exclude single tests somehow).

I assume there's a flag which says "use the intermittant file" for CC  
use

geir


>
> Thanks,
> Mikhail
>
>>
>> geir
>>
>> >
>> > Thanks,
>> > Mikhail
>> >
>> >>
>> >> >
>> >> > I'm for separating these x-lists: we may get rid of CC failures,
>> >> and
>> >> > at the same time
>> >> > those tests will still find regressions (when run pre-commit  
>> tests)
>> >> >
>> >> > Thanks,
>> >> > Mikhail
>> >> >
>> >> >
>> >> >>
>> >> >> >
>> >> >> >>
>> >> >> >> > Any file in this chain may be skipped. Final exclude
lists
>> >> are
>> >> >> >> > generated at
>> >> >> >> > the build time and stored to the ${hy.hdk}/build
 
>> directory.
>> >> >> Thanks
>> >> >> >> > to Alexei
>> >> >> >> > Zakharov for this changes.
>> >> >> >>
>> >> >> >> Another quick q - why not just glom things together in
 
>> memory?
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > If we will run tests against HDK it will nice to skip  
>> excluded
>> >> >> > tests. For
>> >> >> > this reason exclude lists should be stored somewhere in the
>> >> built
>> >> >> > product.
>> >> >> >
>> >> >>
>> >> >> That makes perfect sense.  Thanks
>> >> >>
>> >> >> geir
>> >> >>
>> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > But I have one more question to discuss: should we
use the
>> >> tests
>> >> >> >> from
>> >> >> >> > 'intermittently failed' exclude list for preintegration
>> >> testing?
>> >> >> >>
>> >> >> >> What does that mean?
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > OK. Now pre-integration test include 'ant test' with
>> >> requirements
>> >> >> > that all
>> >> >> > tests should be passed. While we have intermittently failed
>> >> >> tests this
>> >> >> > target sometimes report 'failed' status for these tests  
>> without
>> >> >> > correlation
>> >> >> > with commit changes. We have 2 options here:
>> >> >> >
>> >> >> > - exclude these tests in the platform-specific ex-list
>> >> >> >
>> >> >> > - exclude these tests in the special ex-list
>> >> >> >
>> >> >> > Each option has pro and contra: in the first case we can miss
>> >> the
>> >> >> > regression
>> >> >> > when intermittently failed test became always failed or
>> >> delete test
>> >> >> > from
>> >> >> > exclude lists if it passed only one time.
>> >> >> >
>> >> >> > In the second case we have some overhead to support special
>> >> exclude
>> >> >> > lists.
>> >> >> >
>> >> >> > Note to detect regression the 'default' mode should be 'on
 
>> (run)
>> >> >> > intermittently failed tests' to test commit changes and  
>> 'off' to
>> >> >> > run tests
>> >> >> > under CC.
>> >> >> >
>> >> >> > I'll add switches to on/off levels for resulting exclude 

>> lists.
>> >> >> >
>> >> >> > thanks, Vladimir
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >> > If we use
>> >> >> >> > it we may miss some regression when intermittently
failed
>> >> >> test will
>> >> >> >> > failed
>> >> >> >> > constantly however if we does not use it we need
to run  
>> test
>> >> >> twice
>> >> >> >> > sometimes.
>> >> >> >> > What is correct behavior?
>> >> >> >> >
>> >> >> >> >  thanks, Vladimir
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On 1/28/07, Alex Blewitt <alex.blewitt@gmail.com>
wrote:
>> >> >> >> >>
>> >> >> >> >> Yeah, +1 for using common exclude lists. It makes
it  
>> easier
>> >> >> when
>> >> >> >> >> Harmony gets ported to other operating systems.
And I
>> >> don't see
>> >> >> >> the
>> >> >> >> >> benefit of having empty lists in that case; and
if  
>> nothing's
>> >> >> >> failing,
>> >> >> >> >> you don't need an empty list either :-)
>> >> >> >> >>
>> >> >> >> >> Alex.
>> >> >> >> >>
>> >> >> >> >> On 28/01/07, Alexey Petrenko  
>> <alexey.a.petrenko@gmail.com>
>> >> >> wrote:
>> >> >> >> >> > +1 from me for using common exclude lists
and removing
>> >> empty
>> >> >> >> lists.
>> >> >> >> >> >
>> >> >> >> >> > SY, Alexey
>> >> >> >> >> >
>> >> >> >> >> > 2007/1/16, Alexey Varlamov  
>> <alexey.v.varlamov@gmail.com>:
>> >> >> >> >> > > Folks,
>> >> >> >> >> > >
>> >> >> >> >> > > I've some concerns with recent updates
for x-list
>> >> >> support: can
>> >> >> >> >> we use
>> >> >> >> >> > > a common macros instead of copy-pasting
4 new targets
>> >> into
>> >> >> >> >> build.xml
>> >> >> >> >> > > for each module?
>> >> >> >> >> > > Or, if we can neglect creating a compiled
x-file
>> >> >> >> >> > > (${hy.hdk}/build/<module>.exclude),
just use "if"
>> >> >> attribute of
>> >> >> >> >> > > <excludesfile>, like this:
>> >> >> >> >> > >
>> >> >> >> >> > > <available property="x.list.exist"
>> >> >> >> >> > > file="exclude.${hy.platform}.${hy.test.vm.name}"/>
>> >> >> >> >> > > ...
>> >> >> >> >> > >             <batchtest>
>> >> >> >> >> > >                 <fileset dir="${src.test.java}">
>> >> >> >> >> > >                     <include name="**/*Test.java"/>
>> >> >> >> >> > >                     <excludesfile
>> >> name="exclude.common"/>
>> >> >> >> >> > >                     <excludesfile
name="$ 
>> {exclude.file}
>> >> >> if="
>> >> >> >> >> x.list.exist" />
>> >> >> >> >> > >                 </fileset>
>> >> >> >> >> > >             </batchtest>
>> >> >> >> >> > >
>> >> >> >> >> > > Also, I suggest to delete empty x-list
remained after
>> >> >> >> introducing
>> >> >> >> >> common lists.
>> >> >> >> >> > >
>> >> >> >> >> > > Another issue is with "hy.test.vm.name",
it was a
>> >> >> surprise for
>> >> >> >> >> me that
>> >> >> >> >> > > it is not autodetected yet. Most obvious
way to get
>> >> it is
>> >> >> >> to read
>> >> >> >> >> > > "java.vm.name" property, this only
requires running
>> >> trivial
>> >> >> >> test.
>> >> >> >> >> > >
>> >> >> >> >> > > --
>> >> >> >> >> > > Alexey
>> >> >> >> >> > >
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>


Mime
View raw message