harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [testing] test exclude list: can't we have incremental exclusions?
Date Mon, 27 Nov 2006 13:37:08 GMT


Ivanov, Alexey A wrote:
>> -----Original Message-----
>> From: Geir Magnusson Jr. [mailto:geir@pobox.com]

>> Wouldn't the CI systems running on those platforms catch it automatically?
> 
> That's what we talk about, isn't it? If someone has fixed a test, it's better to exclude
it from *all* the exclude lists and hoping it won't fail on all platforms. If it fails on
a platform, everybody will be notified about the failure. Then we decide either to fix it,
if it's easy and straightforward, or to add this test into exclude list for that certain platform.
> 
> Am I right?

I think we've gone off track.  I think we have enough degrees of freedom 
that we can test a few ways of doing this, and find out what works best. 
  I'm assuming that even if we do the "time waster" process  - remove 
from exclude lists that you can test, and then send a note to the list - 
it's not that big of a burden given we all Know for Certain :) that 
there will be very few of these, if any ;)

geir

> 
> 
> And remove a test from all exclude lists means you need to edit 8 files rather than just
one.
> 
> 
> Regards,
> Alexey.
> 
> 
>> geir
>>
>>> Thanks,
>>>
>>> 24.11.06, Geir Magnusson Jr.<geir@pobox.com> написал(а):
>>>>
>>>> Alexei Zakharov wrote:
>>>>>> Could be - I don't see where the waste of time comes in.
>>>>> Just a simple arithmetic: 8 (test runs) is bigger (longer) than 1 or
>>>>> 2,  + synchronizing costs..
>>>> Sorry?  The 8 files reflect 8 *different platforms*, which are each
>>>> going to run anyway.
>>>>
>>>> geir
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>> 24.11.06, Geir Magnusson Jr.<geir@pobox.com> написал(а):
>>>>>>
>>>>>> Alexei Zakharov wrote:
>>>>>>> In this case we can run against the situation when the test will
>>>>>>> remain excluded forever for some platform.
>>>>>> Why?  Our goal is to get rid of any excluded tests.   Consider the
>>>>>> excluded tests as the existence of a regression, and we should work
>> to
>>>>>> fix ASAP.
>>>>>>
>>>>>> We aren't "parking" them there to be ignored - we're putting them
>>>> there
>>>>>> so that the build can complete while we work on the fixes.  We're
in
>>>>>> essence "grandfathering-in" these regressions.
>>>>>>
>>>>>>> People have their own
>>>>>>> problems. IMHO we should do this only if there are suspicions
>>>> that the
>>>>>>> test is platform dependent. Otherwise it is a waste of time IMO
>>>> and CC
>>>>>>> can handle this.
>>>>>> Could be - I don't see where the waste of time comes in.
>>>>>>
>>>>>> geir
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> 24.11.06, Geir Magnusson Jr.<geir@pobox.com> написал(а):
>>>>>>>> Why not just remove from the platforms you have, and ask
others to
>>>>>>>> update (or not) platforms you don't have as appropriate?
>>>>>>>>
>>>>>>>> geir
>>>>>>>>
>>>>>>>> Tim Ellison wrote:
>>>>>>>>> Alexei Zakharov wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>> So we just should choice what is better: to break
sometimes
>>>>>> tests run
>>>>>>>>>>> or to
>>>>>>>>>>> forget enable test(s) on some platforms.
>>>>>>>>>> Yesterday, when I was removing one of the beans tests
from
>>>> exclude
>>>>>>>>>> lists, I feel a bit uncomfortable while updating
>>>>>>>>>> exclude.linux.x86_64.xxx since I have no (easy) access
to such
>>>>>> systems
>>>>>>>>>> and had no plans to run tests on it. IMHO (in the
perfect world)
>>>>>> the
>>>>>>>>>> fact that I remove or add something from / to
>>>> exclude.linux.x86_64
>>>>>>>>>> means I've at least ran tests for this platform and
obtained
>>>> some
>>>>>>>>>> result. So let's have a common list,  it's easier
to deal
>>>> with it
>>>>>>>>>> psychologically.  :-)
>>>>>>>>>>
>>>>>>>>>> As for above question: +1 for being optimists, i.e.
to remove
>>>>>> the test
>>>>>>>>>> from common list if it passes on all platforms available
to
>>>> tester.
>>>>>>>>> +1 (otherwise I'll share the psychotherapy costs with
you :-)
>>>
> 
> --
> Alexey A. Ivanov
> Intel Enterprise Solutions Software Division

Mime
View raw message