harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [classlib][build] exclude list impl issues
Date Tue, 30 Jan 2007 08:12:45 GMT

On Jan 30, 2007, at 7:40 AM, Mikhail Loenko wrote:

> 2007/1/30, Geir Magnusson Jr. <geir@pobox.com>:
>>
>> On Jan 30, 2007, at 5:17 AM, Mikhail Loenko wrote:
>>
>> > 2007/1/29, Geir Magnusson Jr. <geir@pobox.com>:
>> >>
>> >> On Jan 29, 2007, at 8:45 AM, Vladimir Ivanov wrote:
>> >>
>> >> > On 1/29/07, Geir Magnusson Jr. <geir@pobox.com> wrote:
>> >> >>
>> >> >>
>> >> >> On Jan 28, 2007, at 11:28 PM, Vladimir Ivanov wrote:
>> >> >>
>> >> >> > OK. Current changes for exclude lists were integrated and
 
>> now we
>> >> >> have
>> >> >> > 3-level exclude list:
>> >> >> >
>> >> >> > exlude.common - tests, that failed over all platforms
>> >> >> >
>> >> >> > exclude.<platform> - tests, that failed over specific
>> >> platform only
>> >> >> >
>> >> >> > exclude.<platform>.interm - tests that failed time to
time  
>> over
>> >> >> > specific
>> >> >> > platform only.
>> >> >>
>> >> >> Quick q - why separate interm out?  why not just put in the
>> >> platform
>> >> >> file?
>> >> >
>> >> >
>> >> > To do exclude lists clean up process easier. To delete test from
>> >> the
>> >> > platform list you should check that this test passed one  
>> time. To
>> >> > delete
>> >> > test from 'intermittently failed' exclude list each test  
>> should be
>> >> > run in
>> >> > cycle.
>> >>
>> >> :/
>> >>
>> >> Doesn't the first case simply mean you got lucky?
>> >
>> > Well if some permanently failing tests starts to pass then it's
>> > probably more than
>> > just luck
>>
>> Of course - someone did work (although historically we know that
>> we're not always sure why)
>>
>> A ha.  You're suggesting a test can't be on both lists?  That was the
>> logical conundrum - if you had the same test on both lists, then it's
>> clear it passes *sometimes*.
>
> Well, if the test passes sometimes, then let's don't have such  
> tests :)

I hope that's a joke, along the lines of :

Patient : It hurts when I lift my arm
Doctor : Then don't lift your arm.

> Probably it should be in the "permanent" list with a proper comment

That was what I was getting at - if there is a test that is  
intermittent, then IMO it's a failure, and should be on the main list  
(with a comment, maybe, and a JIRA to track it).  Therefore, I don't  
understand what tests you'd then need to put on the "intermittent" list.

See why I'm confused?

geir

>
> Thanks,
> Mikhail
>
>>
>> >
>> > I'm for separating these x-lists: we may get rid of CC failures,  
>> and
>> > at the same time
>> > those tests will still find regressions (when run pre-commit tests)
>> >
>> > Thanks,
>> > Mikhail
>> >
>> >
>> >>
>> >> >
>> >> >>
>> >> >> > Any file in this chain may be skipped. Final exclude lists
 
>> are
>> >> >> > generated at
>> >> >> > the build time and stored to the ${hy.hdk}/build directory.
>> >> Thanks
>> >> >> > to Alexei
>> >> >> > Zakharov for this changes.
>> >> >>
>> >> >> Another quick q - why not just glom things together in memory?
>> >> >
>> >> >
>> >> >
>> >> > If we will run tests against HDK it will nice to skip excluded
>> >> > tests. For
>> >> > this reason exclude lists should be stored somewhere in the  
>> built
>> >> > product.
>> >> >
>> >>
>> >> That makes perfect sense.  Thanks
>> >>
>> >> geir
>> >>
>> >> >>
>> >> >> >
>> >> >> >
>> >> >> > But I have one more question to discuss: should we use the
 
>> tests
>> >> >> from
>> >> >> > 'intermittently failed' exclude list for preintegration  
>> testing?
>> >> >>
>> >> >> What does that mean?
>> >> >
>> >> >
>> >> >
>> >> > OK. Now pre-integration test include 'ant test' with  
>> requirements
>> >> > that all
>> >> > tests should be passed. While we have intermittently failed
>> >> tests this
>> >> > target sometimes report 'failed' status for these tests without
>> >> > correlation
>> >> > with commit changes. We have 2 options here:
>> >> >
>> >> > - exclude these tests in the platform-specific ex-list
>> >> >
>> >> > - exclude these tests in the special ex-list
>> >> >
>> >> > Each option has pro and contra: in the first case we can miss  
>> the
>> >> > regression
>> >> > when intermittently failed test became always failed or  
>> delete test
>> >> > from
>> >> > exclude lists if it passed only one time.
>> >> >
>> >> > In the second case we have some overhead to support special  
>> exclude
>> >> > lists.
>> >> >
>> >> > Note to detect regression the 'default' mode should be 'on (run)
>> >> > intermittently failed tests' to test commit changes and 'off' to
>> >> > run tests
>> >> > under CC.
>> >> >
>> >> > I'll add switches to on/off levels for resulting exclude lists.
>> >> >
>> >> > thanks, Vladimir
>> >> >
>> >> >
>> >> >
>> >> >> > If we use
>> >> >> > it we may miss some regression when intermittently failed
>> >> test will
>> >> >> > failed
>> >> >> > constantly however if we does not use it we need to run test
>> >> twice
>> >> >> > sometimes.
>> >> >> > What is correct behavior?
>> >> >> >
>> >> >> >  thanks, Vladimir
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On 1/28/07, Alex Blewitt <alex.blewitt@gmail.com> wrote:
>> >> >> >>
>> >> >> >> Yeah, +1 for using common exclude lists. It makes it easier
>> >> when
>> >> >> >> Harmony gets ported to other operating systems. And I
 
>> don't see
>> >> >> the
>> >> >> >> benefit of having empty lists in that case; and if nothing's
>> >> >> failing,
>> >> >> >> you don't need an empty list either :-)
>> >> >> >>
>> >> >> >> Alex.
>> >> >> >>
>> >> >> >> On 28/01/07, Alexey Petrenko <alexey.a.petrenko@gmail.com>
>> >> wrote:
>> >> >> >> > +1 from me for using common exclude lists and removing
 
>> empty
>> >> >> lists.
>> >> >> >> >
>> >> >> >> > SY, Alexey
>> >> >> >> >
>> >> >> >> > 2007/1/16, Alexey Varlamov <alexey.v.varlamov@gmail.com>:
>> >> >> >> > > Folks,
>> >> >> >> > >
>> >> >> >> > > I've some concerns with recent updates for x-list
>> >> support: can
>> >> >> >> we use
>> >> >> >> > > a common macros instead of copy-pasting 4 new
targets  
>> into
>> >> >> >> build.xml
>> >> >> >> > > for each module?
>> >> >> >> > > Or, if we can neglect creating a compiled x-file
>> >> >> >> > > (${hy.hdk}/build/<module>.exclude), just
use "if"
>> >> attribute of
>> >> >> >> > > <excludesfile>, like this:
>> >> >> >> > >
>> >> >> >> > > <available property="x.list.exist"
>> >> >> >> > > file="exclude.${hy.platform}.${hy.test.vm.name}"/>
>> >> >> >> > > ...
>> >> >> >> > >             <batchtest>
>> >> >> >> > >                 <fileset dir="${src.test.java}">
>> >> >> >> > >                     <include name="**/*Test.java"/>
>> >> >> >> > >                     <excludesfile  
>> name="exclude.common"/>
>> >> >> >> > >                     <excludesfile name="${exclude.file}
>> >> if="
>> >> >> >> x.list.exist" />
>> >> >> >> > >                 </fileset>
>> >> >> >> > >             </batchtest>
>> >> >> >> > >
>> >> >> >> > > Also, I suggest to delete empty x-list remained
after
>> >> >> introducing
>> >> >> >> common lists.
>> >> >> >> > >
>> >> >> >> > > Another issue is with "hy.test.vm.name", it
was a
>> >> surprise for
>> >> >> >> me that
>> >> >> >> > > it is not autodetected yet. Most obvious way
to get  
>> it is
>> >> >> to read
>> >> >> >> > > "java.vm.name" property, this only requires
running  
>> trivial
>> >> >> test.
>> >> >> >> > >
>> >> >> >> > > --
>> >> >> >> > > Alexey
>> >> >> >> > >
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>


Mime
View raw message