harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [classlib][build] exclude list impl issues
Date Tue, 30 Jan 2007 06:39:04 GMT

On Jan 30, 2007, at 5:17 AM, Mikhail Loenko wrote:

> 2007/1/29, Geir Magnusson Jr. <geir@pobox.com>:
>>
>> On Jan 29, 2007, at 8:45 AM, Vladimir Ivanov wrote:
>>
>> > On 1/29/07, Geir Magnusson Jr. <geir@pobox.com> wrote:
>> >>
>> >>
>> >> On Jan 28, 2007, at 11:28 PM, Vladimir Ivanov wrote:
>> >>
>> >> > OK. Current changes for exclude lists were integrated and now we
>> >> have
>> >> > 3-level exclude list:
>> >> >
>> >> > exlude.common - tests, that failed over all platforms
>> >> >
>> >> > exclude.<platform> - tests, that failed over specific  
>> platform only
>> >> >
>> >> > exclude.<platform>.interm - tests that failed time to time over
>> >> > specific
>> >> > platform only.
>> >>
>> >> Quick q - why separate interm out?  why not just put in the  
>> platform
>> >> file?
>> >
>> >
>> > To do exclude lists clean up process easier. To delete test from  
>> the
>> > platform list you should check that this test passed one time. To
>> > delete
>> > test from 'intermittently failed' exclude list each test should be
>> > run in
>> > cycle.
>>
>> :/
>>
>> Doesn't the first case simply mean you got lucky?
>
> Well if some permanently failing tests starts to pass then it's
> probably more than
> just luck

Of course - someone did work (although historically we know that  
we're not always sure why)

A ha.  You're suggesting a test can't be on both lists?  That was the  
logical conundrum - if you had the same test on both lists, then it's  
clear it passes *sometimes*.

>
> I'm for separating these x-lists: we may get rid of CC failures, and
> at the same time
> those tests will still find regressions (when run pre-commit tests)
>
> Thanks,
> Mikhail
>
>
>>
>> >
>> >>
>> >> > Any file in this chain may be skipped. Final exclude lists are
>> >> > generated at
>> >> > the build time and stored to the ${hy.hdk}/build directory.  
>> Thanks
>> >> > to Alexei
>> >> > Zakharov for this changes.
>> >>
>> >> Another quick q - why not just glom things together in memory?
>> >
>> >
>> >
>> > If we will run tests against HDK it will nice to skip excluded
>> > tests. For
>> > this reason exclude lists should be stored somewhere in the built
>> > product.
>> >
>>
>> That makes perfect sense.  Thanks
>>
>> geir
>>
>> >>
>> >> >
>> >> >
>> >> > But I have one more question to discuss: should we use the tests
>> >> from
>> >> > 'intermittently failed' exclude list for preintegration testing?
>> >>
>> >> What does that mean?
>> >
>> >
>> >
>> > OK. Now pre-integration test include 'ant test' with requirements
>> > that all
>> > tests should be passed. While we have intermittently failed  
>> tests this
>> > target sometimes report 'failed' status for these tests without
>> > correlation
>> > with commit changes. We have 2 options here:
>> >
>> > - exclude these tests in the platform-specific ex-list
>> >
>> > - exclude these tests in the special ex-list
>> >
>> > Each option has pro and contra: in the first case we can miss the
>> > regression
>> > when intermittently failed test became always failed or delete test
>> > from
>> > exclude lists if it passed only one time.
>> >
>> > In the second case we have some overhead to support special exclude
>> > lists.
>> >
>> > Note to detect regression the 'default' mode should be 'on (run)
>> > intermittently failed tests' to test commit changes and 'off' to
>> > run tests
>> > under CC.
>> >
>> > I'll add switches to on/off levels for resulting exclude lists.
>> >
>> > thanks, Vladimir
>> >
>> >
>> >
>> >> > If we use
>> >> > it we may miss some regression when intermittently failed  
>> test will
>> >> > failed
>> >> > constantly however if we does not use it we need to run test  
>> twice
>> >> > sometimes.
>> >> > What is correct behavior?
>> >> >
>> >> >  thanks, Vladimir
>> >> >
>> >> >
>> >> >
>> >> > On 1/28/07, Alex Blewitt <alex.blewitt@gmail.com> wrote:
>> >> >>
>> >> >> Yeah, +1 for using common exclude lists. It makes it easier  
>> when
>> >> >> Harmony gets ported to other operating systems. And I don't see
>> >> the
>> >> >> benefit of having empty lists in that case; and if nothing's
>> >> failing,
>> >> >> you don't need an empty list either :-)
>> >> >>
>> >> >> Alex.
>> >> >>
>> >> >> On 28/01/07, Alexey Petrenko <alexey.a.petrenko@gmail.com>
 
>> wrote:
>> >> >> > +1 from me for using common exclude lists and removing empty
>> >> lists.
>> >> >> >
>> >> >> > SY, Alexey
>> >> >> >
>> >> >> > 2007/1/16, Alexey Varlamov <alexey.v.varlamov@gmail.com>:
>> >> >> > > Folks,
>> >> >> > >
>> >> >> > > I've some concerns with recent updates for x-list  
>> support: can
>> >> >> we use
>> >> >> > > a common macros instead of copy-pasting 4 new targets
into
>> >> >> build.xml
>> >> >> > > for each module?
>> >> >> > > Or, if we can neglect creating a compiled x-file
>> >> >> > > (${hy.hdk}/build/<module>.exclude), just use "if"
 
>> attribute of
>> >> >> > > <excludesfile>, like this:
>> >> >> > >
>> >> >> > > <available property="x.list.exist"
>> >> >> > > file="exclude.${hy.platform}.${hy.test.vm.name}"/>
>> >> >> > > ...
>> >> >> > >             <batchtest>
>> >> >> > >                 <fileset dir="${src.test.java}">
>> >> >> > >                     <include name="**/*Test.java"/>
>> >> >> > >                     <excludesfile name="exclude.common"/>
>> >> >> > >                     <excludesfile name="${exclude.file}
 
>> if="
>> >> >> x.list.exist" />
>> >> >> > >                 </fileset>
>> >> >> > >             </batchtest>
>> >> >> > >
>> >> >> > > Also, I suggest to delete empty x-list remained after
>> >> introducing
>> >> >> common lists.
>> >> >> > >
>> >> >> > > Another issue is with "hy.test.vm.name", it was a  
>> surprise for
>> >> >> me that
>> >> >> > > it is not autodetected yet. Most obvious way to get it
is
>> >> to read
>> >> >> > > "java.vm.name" property, this only requires running trivial
>> >> test.
>> >> >> > >
>> >> >> > > --
>> >> >> > > Alexey
>> >> >> > >
>> >> >> >
>> >> >>
>> >>
>> >>
>>
>>


Mime
View raw message